date:20200729

Re: Review Request 72713: ATLAS-3892 : Basic search enhanced Feature - Search history

2020-07-29 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72713/#review221417
---




repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Line 526 (original), 534 (patched)


Block #534 - #557 is executed only for TIME_RANGE operator. This doesn't 
look right, given this block was executed for all operators before thisi patch. 
Please review and update.



repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
Lines 797 (patched)


Consider moving #797, #798, #807 - #815 under a  "case 
AtlasBaseTypeDef.ATLAS_TYPE_DATE:", like:

   case AtlasBaseTypeDef.ATLAS_TYPE_DATE:
 if (op == SearchParameters.Operator.TIME_RANGE && 
StringUtils.isNotEmpty(attrVal)) {
   String[] parts  = attrVal.split(ATTRIBUTE_VALUE_DELIMITER);
   String   rangeStart = parts.length > 0 ? parts[0];
   String   rangeEnd   = parts.length > 1 ? parts[1];
   
   attrValue  = StringUtils.isEmpty(rangeStart) ? null : 
Long.parseLong(rangeStart);
   attrValue2 = StringUtils.isEmpty(rangeEnd)   ? null : 
Long.parseLong(rangeEnd);
 } else {
   attrValue  = StringUtils.isEmpty(attrVal) ? null : 
Long.parseLong(attrVal);
 }



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 180 (patched)


getBETPredicate() => getInRangePredicate()



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 188 (patched)


'final' for argument is not useful, and is a distraction in reading the 
code. Please remove.



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 189 (patched)


Instead of returning null, I suggest:
  return generatePredicate(attrName, attrVal, attrVal, attrClass);



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 195 (patched)


add "attrVal2 == null" as well.



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Line 767 (original), 800 (patched)


Interfaces can have 'default' methods, so there is no need to change 
VertexAttributePredicateGenerator to an abstract class.



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 803 (patched)


Consider renaming the new method as generateInRangePredicate()



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 1008 (patched)


Mark 'value2' as final, and initialize to null in constructors at #1010 and 
#1016.



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 1021 (patched)


value  => rangeStart
value2 => rangeEnd



repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
Lines 1075 (patched)


getBETPredicate() => getInRangePredicate()


- Madhan Neethiraj


On July 30, 2020, 5:57 a.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72713/
> ---
> 
> (Updated July 30, 2020, 5:57 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3892
> https://issues.apache.org/jira/browse/ATLAS-3892
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1.This patch implements the search history feature where from UI we will 
> receive below filters.
> 2.In backend we have introduced a new operator "BETWEEN" where all the below 
> 7 operators will be converted to BETWEEN operator.
> 3.There will be a slight change in the JSON where previously we were passing 
> the Attribute value in Search parameters from the UI but for below filters   
> we will not pass any attributeValue except for "custom range" filter we 
> provide attribute value 
>   for eg ;"attributeValue":"159354180,159544260" .
> 
> New 7 filters :
> 
> 1.last 7 days.
> 2.Last month.
> 3.Last 30 days.
> 4.Today.
> 5.Yesterday.
> 6.Custom Range.
> 7.This Month.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java a71787bc0 
>   intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
> fcc4494f2 
>   reposi

Re: Review Request 72713: ATLAS-3892 : Basic search enhanced Feature - Search history

2020-07-29 Thread chaitali


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72713/
---

(Updated July 30, 2020, 5:57 a.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-3892
https://issues.apache.org/jira/browse/ATLAS-3892


Repository: atlas


Description
---

1.This patch implements the search history feature where from UI we will 
receive below filters.
2.In backend we have introduced a new operator "BETWEEN" where all the below 7 
operators will be converted to BETWEEN operator.
3.There will be a slight change in the JSON where previously we were passing 
the Attribute value in Search parameters from the UI but for below filters   we 
will not pass any attributeValue except for "custom range" filter we provide 
attribute value 
  for eg ;"attributeValue":"159354180,159544260" .

New 7 filters :

1.last 7 days.
2.Last month.
3.Last 30 days.
4.Today.
5.Yesterday.
6.Custom Range.
7.This Month.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java a71787bc0 
  intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 
fcc4494f2 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
c9a605355 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 
  
repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java
 8e42d1700 


Diff: https://reviews.apache.org/r/72713/diff/4/

Changes: https://reviews.apache.org/r/72713/diff/3-4/


Testing
---

Tested with below json: 
eg : 
{
  "excludeDeletedEntities": true,
  "includeSubClassifications": true,
  "includeSubTypes": true,
  "includeClassificationAttributes": true,
  "entityFilters": {
"condition": "AND",
"criterion": [
  {
"attributeName": "__timestamp",
"operator": "THISMONTH",
"attributeValue": ""
  }
]
  },
  "tagFilters": null,
  "attributes": [
"__timestamp"
  ],
  "limit": 25,
  "offset": 0,
  "typeName": "Table",
  "classification": null,
  "termName": null
}
eg: Only For custom range filter we provide attribute value.
{
  "excludeDeletedEntities": true,
  "includeSubClassifications": true,
  "includeSubTypes": true,
  "includeClassificationAttributes": true,
  "entityFilters": {
"condition": "AND",
"criterion": [
  {
"attributeName": "createTime",
"operator": "CUSTOM_RANGE",
"attributeValue": "159354180,159544260"
  }
]
  },
  "tagFilters": null,
  "attributes": [
"createTime"
  ],
  "limit": 25,
  "offset": 0,
  "typeName": "Table",
  "classification": null,
  "termName": null
}

* Added Test cases to check new processDateRange method functionality


Thanks,

chaitali

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-29 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/#review221416
---




intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 539 (patched)


I suggest to move this method to AtlasAttribute:

  class AtlasAttribute {
...

public AtlasEntityType getReferencedEntityType(AtlasTypeRegistry 
typeRegistry) {
  return 
getReferencedEntityType(typeRegistry.getType(attributeDef.getTypeName()))
}
  }


- Madhan Neethiraj


On July 29, 2020, 1:41 p.m., Pinal Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72646/
> ---
> 
> (Updated July 29, 2020, 1:41 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3876
> https://issues.apache.org/jira/browse/ATLAS-3876
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Issue:**
> Relationship api doesn't provide approximate count of the related entities in 
> the response.
> 
> **Workaround:**
> Get the total count of related given entity , irrespective of the 
> offset/limit.
> Added 'getApproximateCount' as queryParam in Rest api.
> 
> 
> Also this patch includes **improvement in the time taken to fetch related 
> entities**  .
> Average time taken for the Api to search relationship entities having **5000 
> end vertices** with limit **500**:
> Before: 9seconds
> After applying this patch : 3seconds
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
>   
> repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java
>  86572598a 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  4b9564295 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  863a00350 
>   repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
> 5069d78c8 
>   webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 
> 
> 
> Diff: https://reviews.apache.org/r/72646/diff/3/
> 
> 
> Testing
> ---
> 
> Manually tested
> Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
> (Failed in Impala build)
> 
> 
> Thanks,
> 
> Pinal Shah
> 
>

Re: Deleted Relationships definition

2020-07-29 Thread Madhan Neethiraj

Where in UI do you see the relationship in active state? Note that if the state 
of the related-entity itself is active, UI would show related entity's name as 
active. Rendering entity name as 'deleted' would be misleading, as the entity 
is still active.

Madhan



On 7/29/20, 3:22 AM, "Verdan Mahmood"  wrote:

I have a question related to the relationships.

Let's say I have two entities User and Table. and I developed a
relationship between them.
If I delete the relationship using the Relationship API, in the UI the
relationship remains the same with no difference in the color or state,
even though the realtionshipStatus is DELETED. Is this expected behaviour
or should that relation be gone from the frontend?

Can someone please confirm if this is expected behaviour or should that
relation be gone from the frontend? or am I missing something?

P.S: I do not want to delete any of the entities.

Best,
*Verdan Mahmood*
M: (+31) 655 576 560

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

2020-07-29 Thread Damian Warszawski

It appears that by setting consistency to lock, there is different field
mapping in freetext handler. I am working to fix that.

On Wed, 29 Jul 2020 at 22:31, Madhan Neethiraj  wrote:

> Damian,
>
> A number of tests fail with this patch - see below. Can you please run
> 'mvn clean package' and investigate these failures?
>
> Thanks,
> Madhan
>
> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 105.055
> sec <<< FAILURE! - in org.apache.atlas.discovery.FreeTextSearchProcessorTest
> searchByNameSortBy(org.apache.atlas.discovery.FreeTextSearchProcessorTest)
> Time elapsed: 0.068 sec  <<< FAILURE!
> java.lang.AssertionError: expected [3] but found [0]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:496)
> at org.testng.Assert.assertEquals(Assert.java:125)
> at org.testng.Assert.assertEquals(Assert.java:267)
> at org.testng.Assert.assertEquals(Assert.java:277)
> at
> org.apache.atlas.discovery.FreeTextSearchProcessorTest.searchByNameSortBy(FreeTextSearchProcessorTest.java:91)
>
> searchTablesByName(org.apache.atlas.discovery.FreeTextSearchProcessorTest)
> Time elapsed: 0.01 sec  <<< FAILURE!
> java.lang.AssertionError: expected [3] but found [0]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:496)
> at org.testng.Assert.assertEquals(Assert.java:12T5e)
> at org.testng.Assert.asssts run: 2, Failures: 0, Errors: 0,
> Skipped: 0, Time elapsed: 99.643 sec - in
> org.apache.atlas.services.MetricsServiceTestertEquals(Assert.java:267)
> at org.testng.Assert.assertEquals(Assert.java:277)
> at
> org.apache.atlas.discovery.FreeTextSearchProcessorTest.searchTablesByName(FreeTextSearchProcessorTest.java:71)
>
> Tests run: 19, Failures: 13, Errors: 0, Skipped: 0, Time elapsed: 107.395
> sec <<< FAILURE! - in org.apache.atlas.discovery.AtlasDiscoveryServiceTest
> query_ALLTag(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)  Time
> elapsed: 0.073 sec  <<< FAILURE!
> java.lang.AssertionError: expected [5] but found [2]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:496)
> at org.testng.Assert.assertEquals(Assert.java:125)
> at org.testng.Assert.assertEquals(Assert.java:372)
> at org.testng.Assert.assertEquals(Assert.java:382)
> at
> org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_ALLTag(AtlasDiscoveryServiceTest.java:125)
>
> query_ALLTag_tagFilter(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)
> Time elapsed: 0.044 sec  <<< FAILURE!
> java.lang.AssertionError: expected [4] but found [1]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:496)
> at org.testng.Assert.assertEquals(Assert.java:125)
> at org.testng.Assert.assertEquals(Assert.java:372)
> at org.testng.Assert.assertEquals(Assert.java:382)
> at
> org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_ALLTag_tagFilter(AtlasDiscoveryServiceTest.java:140)
>
> query_ALLWildcardTag(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)
> Time elapsed: 0.016 sec  <<< FAILURE!
> java.lang.AssertionError: expected [5] but found [2]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:496)
> at org.testng.Assert.assertEquals(Assert.java:125)
> at org.testng.Assert.assertEquals(Assert.java:372)
> at org.testng.Assert.assertEquals(Assert.java:382)
> at
> org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_ALLWildcardTag(AtlasDiscoveryServiceTest.java:165)
>
> query_NOTCLASSIFIEDTag(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)
> Time elapsed: 0.026 sec  <<< FAILURE!
> java.lang.AssertionError: expected [true] but found [false]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:496)
> at org.testng.Assert.assertTrue(Assert.java:42)
> at org.testng.Assert.assertTrue(Assert.java:52)
> at
> org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_NOTCLASSIFIEDTag(AtlasDiscoveryServiceTest.java:151)
>
> query_entity(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)  Time
> elapsed: 0.036 sec  <<< FAILURE!
> java.lang.AssertionError: expected [true] but found [false]
> at org.testng.Assert.fail(Assert.java:94)
> at org.testng.Assert.failNotEquals(Assert.java:496)
> at org.testng.Assert.assertTrue(Assert.java:42)
> at org.testng.Assert.assertTrue(Assert.java:52)
> at
> org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_entity(AtlasDiscoveryServiceTest.java:214)
>
> query_entity_entityFilter(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)
> Time elapsed: 0.029 sec  <<< FAILURE!
> java.lang.AssertionError: expected [true] but found [false]
>

Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-29 Thread Ashutosh Mestry via Review Board



> On July 23, 2020, 9:16 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
> > Lines 409 (patched)
> > 
> >
> > should we reset/clear these values for EntityAuditEventV2? It might 
> > have remnant values when reused again?

This has been addressed by clearing the unused elements from the list.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221337
---


On July 29, 2020, 6:23 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 29, 2020, 6:23 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/model/Clearable.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> 63116d467 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationSender.java
>  c2ae5122b 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/9/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2067/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

2020-07-29 Thread Madhan Neethiraj

Damian,

A number of tests fail with this patch - see below. Can you please run 'mvn 
clean package' and investigate these failures?

Thanks,
Madhan

Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 105.055 sec <<< 
FAILURE! - in org.apache.atlas.discovery.FreeTextSearchProcessorTest
searchByNameSortBy(org.apache.atlas.discovery.FreeTextSearchProcessorTest)  
Time elapsed: 0.068 sec  <<< FAILURE!
java.lang.AssertionError: expected [3] but found [0]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertEquals(Assert.java:125)
at org.testng.Assert.assertEquals(Assert.java:267)
at org.testng.Assert.assertEquals(Assert.java:277)
at 
org.apache.atlas.discovery.FreeTextSearchProcessorTest.searchByNameSortBy(FreeTextSearchProcessorTest.java:91)

searchTablesByName(org.apache.atlas.discovery.FreeTextSearchProcessorTest)  
Time elapsed: 0.01 sec  <<< FAILURE!
java.lang.AssertionError: expected [3] but found [0]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertEquals(Assert.java:12T5e)
at org.testng.Assert.asssts run: 2, Failures: 0, Errors: 0, Skipped: 0, 
Time elapsed: 99.643 sec - in 
org.apache.atlas.services.MetricsServiceTestertEquals(Assert.java:267)
at org.testng.Assert.assertEquals(Assert.java:277)
at 
org.apache.atlas.discovery.FreeTextSearchProcessorTest.searchTablesByName(FreeTextSearchProcessorTest.java:71)

Tests run: 19, Failures: 13, Errors: 0, Skipped: 0, Time elapsed: 107.395 sec 
<<< FAILURE! - in org.apache.atlas.discovery.AtlasDiscoveryServiceTest
query_ALLTag(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)  Time 
elapsed: 0.073 sec  <<< FAILURE!
java.lang.AssertionError: expected [5] but found [2]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertEquals(Assert.java:125)
at org.testng.Assert.assertEquals(Assert.java:372)
at org.testng.Assert.assertEquals(Assert.java:382)
at 
org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_ALLTag(AtlasDiscoveryServiceTest.java:125)

query_ALLTag_tagFilter(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)  
Time elapsed: 0.044 sec  <<< FAILURE!
java.lang.AssertionError: expected [4] but found [1]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertEquals(Assert.java:125)
at org.testng.Assert.assertEquals(Assert.java:372)
at org.testng.Assert.assertEquals(Assert.java:382)
at 
org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_ALLTag_tagFilter(AtlasDiscoveryServiceTest.java:140)

query_ALLWildcardTag(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)  
Time elapsed: 0.016 sec  <<< FAILURE!
java.lang.AssertionError: expected [5] but found [2]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertEquals(Assert.java:125)
at org.testng.Assert.assertEquals(Assert.java:372)
at org.testng.Assert.assertEquals(Assert.java:382)
at 
org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_ALLWildcardTag(AtlasDiscoveryServiceTest.java:165)

query_NOTCLASSIFIEDTag(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)  
Time elapsed: 0.026 sec  <<< FAILURE!
java.lang.AssertionError: expected [true] but found [false]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertTrue(Assert.java:42)
at org.testng.Assert.assertTrue(Assert.java:52)
at 
org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_NOTCLASSIFIEDTag(AtlasDiscoveryServiceTest.java:151)

query_entity(org.apache.atlas.discovery.AtlasDiscoveryServiceTest)  Time 
elapsed: 0.036 sec  <<< FAILURE!
java.lang.AssertionError: expected [true] but found [false]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertTrue(Assert.java:42)
at org.testng.Assert.assertTrue(Assert.java:52)
at 
org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_entity(AtlasDiscoveryServiceTest.java:214)

query_entity_entityFilter(org.apache.atlas.discovery.AtlasDiscoveryServiceTest) 
 Time elapsed: 0.029 sec  <<< FAILURE!
java.lang.AssertionError: expected [true] but found [false]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertTrue(Assert.java:42)
at org.testng.Assert.assertTrue(Assert.java:52)
at 
org.apache.atlas.discovery.AtlasDiscoveryServiceTest.query_entity_entityFilter(AtlasDiscoveryServiceTe

Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-29 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221413
---



What is the impact on the memory usage improvement, after FixedBufferList use 
is removed from EntityNotificationListenerV2?


intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 41 (patched)


"incrementCapacityBy == 0" -> "incrementCapacityBy <= 0"



repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
Line 278 (original), 269 (patched)


In case of any exception in instanceConverter.getAndCacheEntity(), 'events' 
list might be left with some entries - since events.toList() may not be called. 
And these entries will be used in subsequent event dispatch from this thread.

I suggest the following:
 - add methods:
   class EntityAuditListenerV2 {
 ..
 FixedBufferList getEventsList() {
   FixedBufferList ret = AUDIT_EVENTS_BUFFER.get();
 
   ret.reset();
 
   return ret;
 }
   }

   class FixedBufferList {
  ..
  public void reset() {
for (int i = 0; i < length; i++) {
  buffer.get(i).clear();
}

length = 0;
  }
  
  public List toList() {
return this.buffer.subList(0, this.length); // no need to reset 
length here
  }
   }


- Madhan Neethiraj


On July 29, 2020, 6:23 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 29, 2020, 6:23 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/model/Clearable.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> 63116d467 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationSender.java
>  c2ae5122b 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/9/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2067/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-29 Thread Ashutosh Mestry via Review Board



> On July 24, 2020, 7:14 a.m., Madhan Neethiraj wrote:
> > intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
> > Lines 67 (patched)
> > 
> >
> > incrementCapacityBy is already an instance member, line #29 above. It 
> > is unnecessary to pass it as  parameter to methods at #67, #79. Please 
> > review and update.

I removed the initial capacity from the class altogether.


> On July 24, 2020, 7:14 a.m., Madhan Neethiraj wrote:
> > intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
> > Lines 73 (patched)
> > 
> >
> > This assumes that access is done in strict ascending order sequence, 
> > without any gaps. This is not a good assumption for a generic/reusable 
> > class implementation. This will be broken with the following access pattern:
> >   FixedBufferList list = new FixedBufferList<>(10, 1);
> >   
> >   list.getForUpdate(0);
> >   list.getForUpdate(15);
> > 
> > Please review my earlier comment and update to remove this assumption. 
> > Also, I suggest to collapse following methods into one, named 
> > ensureCapacity():
> >  - request()
> >  - ensureCapacity(): it not obvious what the return value of this 
> > method is, and it is critical for the caller at #74. All this noise, and 
> > confusion can be avoided by simply collapsing this method into previous 
> > method at #67.
> >  - instantiateItems(): this method is not callable from any other 
> > context, as it simply "adds()" to the buffer

I re-worked the interface. There is no way to pass index.


> On July 24, 2020, 7:14 a.m., Madhan Neethiraj wrote:
> > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
> > Line 168 (original), 175 (patched)
> > 
> >
> > sendNotifications() can be called from the same thread multiple times - 
> > for example when a transaction involves create/update/delete of entities 
> > i.e. a graph transaction can call more than one of the following methods:
> > - onEntitiesAdded()
> > - onEntitiesUpdated()
> > - onEntitiesDeleted()
> > - onEntitiesPurged()
> > - onClassificationsAdded()
> > - onClassificationsUpdated()
> > - onClassificationsDeleted()
> > 
> > Each call would end up calling EntityNotificationSender.send() with the 
> > notification objects, which in turn stores the objects in a list. The 
> > second call in the same transaction will end up overwriting 
> > EntityNotificationV2 instances in list buffer - which will cause the 
> > notification objects stored in EntityNotificationSender.send() be updated 
> > as well. This will result in incorrect notification to be sent. Please 
> > review this carefully and update.

I addressed by eliminating usage of FixedBufferList from this class.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221351
---


On July 24, 2020, 5:18 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 24, 2020, 5:18 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a

Re: Review Request 72666: Notification: Solution to Memory Build-up

2020-07-29 Thread Ashutosh Mestry via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/
---

(Updated July 29, 2020, 6:23 p.m.)


Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and 
Sarath Subramanian.


Changes
---

Updates include: Addressed review comments.


Bugs: ATLAS-3878
https://issues.apache.org/jira/browse/ATLAS-3878


Repository: atlas


Description
---

**Background**
See JIRA for details.

*Analysis* Using memory profiling tools, it was observed that large number of 
notification objects were created. These stayed in memory and later were 
promoted to higher generation, thereby taking even longer to be collected.

**Approach**
Using the fixed-buffer approach to address the problem of creating large number 
of small objects.

New *FixedBufferList* This is an encapsulation over *ArrayList*. During initial 
allocation, list is populated with default values. Features:
- Setting of values to these pre-allocated objects is achieved by first doing a 
*get* on the element and then assigning values to it.
- *toList* fetches the sub-list from the encapsulating list. This uses the 
state within the class to fetch the right length for the returning array.

New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
necessary since the list is now part class's state.
Modified *EntityAuditListenerV2* Uses the new classes.
Modifed *EntityNotificationListener* Uses the new classes.

**Verification**
- Using the test setup, the memory usage was observed over a period of 24 hrs. 
- Memory usage and object allocation was obvserved using memory profiler.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
  intg/src/main/java/org/apache/atlas/model/Clearable.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
63116d467 
  intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
  intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
 79527acfa 
  
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationSender.java
 c2ae5122b 


Diff: https://reviews.apache.org/r/72666/diff/9/

Changes: https://reviews.apache.org/r/72666/diff/8-9/


Testing
---

**Unit testing**
Unit tests added for the new classes.

**Volume testing**
Setup:
- Node: Threads 40, Core: 40, Allocated Memory: 12 GB
- Multiple Kafka queues ingesting data.
- Bulk entity creation using custom script ingesting 100M entities.

Memory usage stayed between 0 and 5% during the 24 hr period.

With:
- Workers: 64
- Batch size: 50 (fewer elements in batch improve commit time and audit write 
time).
- Throughput: ~1.2 M entities per hour. Without out of memory error.

**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2067/


Thanks,

Ashutosh Mestry

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-29 Thread Pinal Shah


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

(Updated July 29, 2020, 1:41 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description (updated)
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.
Added 'getApproximateCount' as queryParam in Rest api.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs
-

  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
  
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 
86572598a 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/3/


Testing
---

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
(Failed in Impala build)


Thanks,

Pinal Shah

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

2020-07-29 Thread Pinal Shah


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72646/
---

(Updated July 29, 2020, 1:38 p.m.)


Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Changes
---

Addressed Review comments


Bugs: ATLAS-3876
https://issues.apache.org/jira/browse/ATLAS-3876


Repository: atlas


Description
---

**Issue:**
Relationship api doesn't provide approximate count of the related entities in 
the response.

**Workaround:**
Get the total count of related given entity , irrespective of the offset/limit.


Also this patch includes **improvement in the time taken to fetch related 
entities**  .
Average time taken for the Api to search relationship entities having **5000 
end vertices** with limit **500**:
Before: 9seconds
After applying this patch : 3seconds


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 58de4cc47 
  
repository/src/main/java/org/apache/atlas/discovery/AtlasDiscoveryService.java 
86572598a 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
4b9564295 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 863a00350 
  repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 
5069d78c8 
  webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java cdeb912a4 


Diff: https://reviews.apache.org/r/72646/diff/3/

Changes: https://reviews.apache.org/r/72646/diff/2-3/


Testing
---

Manually tested
Precommit : https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2011 
(Failed in Impala build)


Thanks,

Pinal Shah

[jira] [Updated] (ATLAS-3904) Regression: Glossary term deatils page not able to render the classfication tab

2020-07-29 Thread Keval Bhatt (Jira)



 [ 
https://issues.apache.org/jira/browse/ATLAS-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt updated ATLAS-3904:
---
Attachment: ATLAS-3904.patch

> Regression: Glossary term deatils page not able to render the classfication 
> tab
> ---
>
> Key: ATLAS-3904
> URL: https://issues.apache.org/jira/browse/ATLAS-3904
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: ATLAS-3904.patch, Screen Shot 2020-07-28 at 5.09.08 
> PM.png
>
>
> Click on the term details page and check the console.
>  
> !Screen Shot 2020-07-28 at 5.09.08 PM.png|width=618,height=304!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

2020-07-29 Thread Damian Warszawski



> On July 28, 2020, 10:55 p.m., Madhan Neethiraj wrote:
> > Ship It!

Can I ask you to merge it? I don't think I have permissions to merge it to 
master.


- Damian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72695/#review221401
---


On July 25, 2020, 8:53 p.m., Damian Warszawski wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72695/
> ---
> 
> (Updated July 25, 2020, 8:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Bolke de Bruin, madhan, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Optional configuration to support locks on JanusGraph to ensure data 
> consitency.
> 
> JanusGraph is eventually consistent by default which is efficient but results 
> in duplicates when race condition occurs.
> 
> 
> Reference to jira 
> https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-3398
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
>  6ef9cb76c 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
> 
> 
> Diff: https://reviews.apache.org/r/72695/diff/3/
> 
> 
> Testing
> ---
> 
> Not possible to reproduce the error on local machine. Enable locking on our 
> dev env and have not introduce any regression.
> 
> 
> Thanks,
> 
> Damian Warszawski
> 
>

Deleted Relationships definition

2020-07-29 Thread Verdan Mahmood

I have a question related to the relationships.

Let's say I have two entities User and Table. and I developed a
relationship between them.
If I delete the relationship using the Relationship API, in the UI the
relationship remains the same with no difference in the color or state,
even though the realtionshipStatus is DELETED. Is this expected behaviour
or should that relation be gone from the frontend?

Can someone please confirm if this is expected behaviour or should that
relation be gone from the frontend? or am I missing something?

P.S: I do not want to delete any of the entities.

Best,
*Verdan Mahmood*
M: (+31) 655 576 560

Re: Review Request 72713: ATLAS-3892 : Basic search enhanced Feature - Search history

Re: Review Request 72713: ATLAS-3892 : Basic search enhanced Feature - Search history

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Re: Deleted Relationships definition

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

Re: Review Request 72666: Notification: Solution to Memory Build-up

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

Re: Review Request 72666: Notification: Solution to Memory Build-up

Re: Review Request 72666: Notification: Solution to Memory Build-up

Re: Review Request 72666: Notification: Solution to Memory Build-up

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

Re: Review Request 72646: ATLAS-3876 : Relationship Search API not showing correct approximateCount

[jira] [Updated] (ATLAS-3904) Regression: Glossary term deatils page not able to render the classfication tab

Re: Review Request 72695: Optional configuration to support locks on JanusGraph to ensure data consitency.

Deleted Relationships definition

15 matches

Site Navigation

Mail list logo

Footer information