Re: Review Request 71012: [ATLAS-3310] - Relationships : After updating a bigint attribute , any operation on relationship instance, entity throws 500 internal server exception.

[mayank jain]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71012/
---

(Updated July 9, 2019, 8:39 a.m.)


Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Changes
---

Updated patch to handle bigInteger datatype case for create and update 
relationship
In validate Relation end1 & end2 were going null due to which ValideData was 
false and getNormalize method was not called.
Fixed issue of end1 & end2 going null to validateRelationship method.


Bugs: ATLAS-3310
https://issues.apache.org/jira/browse/ATLAS-3310


Repository: atlas


Description
---

This patch will provide the fix to the BigInteger attributes in relationship.

Working on Unit test case for testing BigInteger case.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java 98071b2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
 54059e8 


Diff: https://reviews.apache.org/r/71012/diff/2/

Changes: https://reviews.apache.org/r/71012/diff/1-2/


Testing
---

Modified the relationship attributes of hive_table and column with the below 
mentioned API :
link used: localhost:21000/api/atlas/v2/types/typedefs
Request Type : PUT
JSON : 
{   
"relationshipDefs": [
{
"category": "RELATIONSHIP",
"guid": "8339e074-8a76-4ab6-b716-2e6a5883c158",
"createdBy": "mayank",
"updatedBy": "mayank",
"createTime": 1561985660068,
"updateTime": 1561985660068,
"version": 1,
"name": "hive_table_columns",
"description": "hive_table_columns",
"typeVersion": "1.2",
"serviceType": "hive",
"attributeDefs": [
{
"name": "description",
"typeName": "string",
"isOptional": true,
"cardinality": "SINGLE",
"valuesMinCount": 0,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": true,
"includeInNotification": false,
"description": "A sample attr for holdig ddescription type 
of attribute",
"searchWeight": -1
},
{
"name": "samplebigint",
"typeName": "biginteger",
"isOptional": true,
"cardinality": "SINGLE",
"valuesMinCount": 0,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": true,
"includeInNotification": false,
"description": "A sample attr for holdig BigInteger type of 
data",
"searchWeight": -1
}
],
"relationshipCategory": "COMPOSITION",
"relationshipLabel": "__hive_table.columns",
"propagateTags": "NONE",
"endDef1": {
"type": "hive_table",
"name": "columns",
"isContainer": true,
"cardinality": "SET",
"isLegacyAttribute": true
},
"endDef2": {
"type": "hive_column",
"name": "table",
"isContainer": false,
"cardinality": "SINGLE",
"isLegacyAttribute": true
}
}
]
}
==
After this  we can create a relationship between a hive table and a hive column 
using below mentioned API request
Now this request provides values for BigInteger Type and the values are been 
succesfully converted to respective types without misbehaving.
Link used : http://localhost:21000/api/atlas/v2/relationship
Request Type : POST
JSON :
{
"end1" :

{ "guid" : "4d5adf00-2c9b-4877-ad23-c41fd7319150" }
,
"end2" :

{ "guid" : "b0d889d1-008f-44ba-85cd-ab43abbb9a00" }
,
"typeName":"hive_table_columns",
"attributes" : {
"description" : "It is a very good description ",
"samplebigint" : 3218

}
}
==


Thanks,

mayank jain

[jira] [Updated] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests

[Nixon Rodrigues (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues updated ATLAS-3317:
---
Attachment: 0001-ATLAS-3317-Set-maxWait-systemProperty-in-jetty-maven.patch

> java.lang.Exception: Timeout scanning annotations while running Integration 
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: 
> 0001-ATLAS-3317-Set-maxWait-systemProperty-in-jetty-maven.patch, maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>  
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @ 
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory = 
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides =  none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at 
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations 
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure 
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure 
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext 
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart 
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart 
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart 
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart 
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty 
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute 
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute 
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute 
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:137)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:210)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:156)
> at org.apache.

[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests

[Nixon Rodrigues (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881069#comment-16881069
 ] 

Nixon Rodrigues commented on ATLAS-3317:


Integration tests are launched after adding higher max timeout value.

https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1243/console

> java.lang.Exception: Timeout scanning annotations while running Integration 
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: 
> 0001-ATLAS-3317-Set-maxWait-systemProperty-in-jetty-maven.patch, maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>  
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @ 
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory = 
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides =  none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at 
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations 
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure 
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure 
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext 
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart 
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart 
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart 
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart 
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty 
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute 
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute 
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute 
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:137)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java

[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests

[Keval Bhatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881081#comment-16881081
 ] 

Keval Bhatt commented on ATLAS-3317:


+1 for the patch [~nixonrodrigues]

> java.lang.Exception: Timeout scanning annotations while running Integration 
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nixon Rodrigues
>Priority: Major
> Attachments: 
> 0001-ATLAS-3317-Set-maxWait-systemProperty-in-jetty-maven.patch, maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>  
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @ 
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory = 
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides =  none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at 
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations 
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure 
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure 
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext 
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart 
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart 
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart 
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart 
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty 
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute 
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute 
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute 
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:137)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:210)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:156)
> at org.ap

[jira] [Assigned] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests

[Nixon Rodrigues (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues reassigned ATLAS-3317:
--

Assignee: Nixon Rodrigues

> java.lang.Exception: Timeout scanning annotations while running Integration 
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Attachments: 
> 0001-ATLAS-3317-Set-maxWait-systemProperty-in-jetty-maven.patch, maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>  
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @ 
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory = 
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides =  none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at 
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations 
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure 
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure 
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext 
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart 
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart 
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart 
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart 
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty 
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute 
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute 
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute 
> (JettyDeployWar.java:65)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
> (DefaultBuildPluginManager.java:137)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:210)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:156)
> at org.apache.mave

[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881086#comment-16881086
 ] 

ASF subversion and git services commented on ATLAS-3317:


Commit 4749e7a1c04ab08ffc5f8d582ca5de3c35b81ecb in atlas's branch 
refs/heads/master from Nixon Rodrigues
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=4749e7a ]

ATLAS-3317 : Set maxWait systemProperty in jetty maven plugin to avoid timeout 
scanning annotations while running Integration Tests


> java.lang.Exception: Timeout scanning annotations while running Integration 
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Attachments: 
> 0001-ATLAS-3317-Set-maxWait-systemProperty-in-jetty-maven.patch, maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>  
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @ 
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory = 
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides =  none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at 
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations 
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure 
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure 
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext 
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart 
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart 
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart 
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart 
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty 
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute 
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute 
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployWar.e

[jira] [Commented] (ATLAS-3317) java.lang.Exception: Timeout scanning annotations while running Integration Tests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881279#comment-16881279
 ] 

ASF subversion and git services commented on ATLAS-3317:


Commit bd5d2da29367a164a505d7e695653fc7dd3ebe40 in atlas's branch 
refs/heads/branch-2.0 from Nixon Rodrigues
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=bd5d2da ]

ATLAS-3317 : Set maxWait systemProperty in jetty maven plugin to avoid timeout 
scanning annotations while running Integration Tests


> java.lang.Exception: Timeout scanning annotations while running Integration 
> Tests
> -
>
> Key: ATLAS-3317
> URL: https://issues.apache.org/jira/browse/ATLAS-3317
> Project: Atlas
>  Issue Type: Bug
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
>Priority: Major
> Attachments: 
> 0001-ATLAS-3317-Set-maxWait-systemProperty-in-jetty-maven.patch, maxWait.diff
>
>
> run :- *mvn clean install -DskipUTs*
>  
> {noformat}
> [INFO] --- jetty-maven-plugin:9.3.14.v20161028:deploy-war (start-jetty) @ 
> atlas-webapp ---
> [INFO] Logging initialized @220653ms
> [INFO] Configuring Jetty for project: Apache Atlas Web Application
> [INFO] Context path = /
> [INFO] Tmp directory = 
> /home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/tmp
> [INFO] Web defaults = org/eclipse/jetty/webapp/webdefault.xml
> [INFO] Web overrides =  none
> [INFO] jetty-9.3.14.v20161028
> [INFO] Scanning elapsed time=67258ms
> [WARNING] Failed startup of context o.e.j.m.p.JettyWebAppContext@2eb2d10c{/,
> [file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE}\{/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war}|file:///home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT/,UNAVAILABLE%7D%7B/home/jenkins/jenkins-slave/workspace/PreCommit-ATLAS-Build-Test@2/webapp/target/atlas-webapp-3.0.0-SNAPSHOT.war%7D]
> java.lang.Exception: Timeout scanning annotations
> at 
> org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations 
> (AnnotationConfiguration.java:578)
> at org.eclipse.jetty.annotations.AnnotationConfiguration.configure 
> (AnnotationConfiguration.java:447)
> at org.eclipse.jetty.webapp.WebAppContext.configure 
> (WebAppContext.java:494)
> at org.eclipse.jetty.webapp.WebAppContext.startContext 
> (WebAppContext.java:1361)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart 
> (ContextHandler.java:778)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart 
> (ServletContextHandler.java:262)
> at org.eclipse.jetty.webapp.WebAppContext.doStart (WebAppContext.java:520)
> at org.eclipse.jetty.maven.plugin.JettyWebAppContext.doStart 
> (JettyWebAppContext.java:398)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart 
> (ContextHandlerCollection.java:161)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:113)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start 
> (ContainerLifeCycle.java:131)
> at org.eclipse.jetty.server.Server.start (Server.java:422)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart 
> (ContainerLifeCycle.java:105)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart 
> (AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart (Server.java:389)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
> (AbstractLifeCycle.java:68)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty 
> (AbstractJettyMojo.java:460)
> at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute 
> (AbstractJettyMojo.java:328)
> at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute 
> (JettyRunWarMojo.java:64)
> at org.eclipse.jetty.maven.plugin.JettyDeployW

Re: Review Request 71025: Import Service: Support Concurrent Ingest

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71025/
---

(Updated July 9, 2019, 4:42 p.m.)


Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and 
Sarath Subramanian.


Changes
---

Updates include:
- Added CURL call and request json.


Bugs: ATLAS-3320
https://issues.apache.org/jira/browse/ATLAS-3320


Repository: atlas


Description (updated)
---

**Approach**
- Use existing producer-consumer (PC) framework.
- Modify _BulkImporterImpl_ to implement _WorkItemConsumer_.
- Add support for configuring number of workers and batch size within 
_AtlasImportRequest_.

_AtlasImportRequest_
```
{
"options": {
"numWorkers": 8,
"batchSize": 25
}
}
```

**CURL**
```
curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
"Cache-Control: no-cache" -F request=@./import-options.json -F 
data=@./Default-3-pre.zip http://localhost:21000/api/atlas/admin/import
```


Diffs
-

  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 499e8d1af 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
0b3ede93f 
  intg/src/main/resources/atlas-log4j.xml 4f74c2abb 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
3ded79842 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
 2f330c093 


Diff: https://reviews.apache.org/r/71025/diff/1/


Testing
---

**Unit tests**
Existing tests.

**Functional tests**
- Verified import for pre-1.0 and post-1.0 exported ZIP files.

**Volume tests**
- Measure performance with large data.

+--+--+--++
| File | Before   | After| Configuration  |
+--+--+--++
| smalldb  |   6 min  |3 min | Shards: 4, Threads: 8  |
| (2.2 MB) |  |  ||
+--+--+--++
| largedb  |3 hrs |  20 mins | Shards: 4, Threads: 16 |
| (40 MB)  |  |  ||
+--+--+--++


Thanks,

Ashutosh Mestry

[jira] [Created] (ATLAS-3321) Introduce atlas metadata namespace

[Sarath Subramanian (JIRA)]

Sarath Subramanian created ATLAS-3321:
-

 Summary: Introduce atlas metadata namespace 
 Key: ATLAS-3321
 URL: https://issues.apache.org/jira/browse/ATLAS-3321
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: Sarath Subramanian
Assignee: Sarath Subramanian
 Fix For: 2.1.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-3321) Introduce atlas metadata namespace

[Sarath Subramanian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3321:
--
Description: 
Atlas has the notion of *clusterName* which is used as a global namespace in 
which you can define the context in which entity instances are defined and 
grouped. A cluster name defines a scope of operation in atlas and is specified 
in parent entity types - _hive_db (Hive), hbase_namespace (HBase), fs_path 
(File System), storm_topology (Storm), spark_db (Spark)_

cluster name is also used to construct _qualifiedName_ attribute in atlas -
 * hive_db - db_name@cluster_name
 * hive_table - db_name.table_name@cluster_name
 * hive_column - db_name.table_name.column_name@cluster_name

Cluster name works for simple environments wherein we have a single cluster and 
single service instances running (single Hive or HBase service). In complex 
deployments - a single cluster may host multiple instances of the same service 
(two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't 
help to provide context to individual service instances and metadata groupings.

So we introduce - *"metadata namespace"* where a single cluster can contain 
multiple metadata namespaces. The qualified name for atlas will be generated 
using metadata namespace name instead of cluster name.

For each hook enabled we specify the property - *"atlas.metadata.namespace"* 
which defines the scope/context of the metadata.

> Introduce atlas metadata namespace 
> ---
>
> Key: ATLAS-3321
> URL: https://issues.apache.org/jira/browse/ATLAS-3321
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 2.1.0
>
>
> Atlas has the notion of *clusterName* which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - _hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)_
> cluster name is also used to construct _qualifiedName_ attribute in atlas -
>  * hive_db - db_name@cluster_name
>  * hive_table - db_name.table_name@cluster_name
>  * hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> So we introduce - *"metadata namespace"* where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> For each hook enabled we specify the property - *"atlas.metadata.namespace"* 
> which defines the scope/context of the metadata.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [atlas] thealmightygrant commented on a change in pull request #61: ATLAS-3295: Remove unused setup code and documentation

[GitBox]

thealmightygrant commented on a change in pull request #61: ATLAS-3295: Remove 
unused setup code and documentation
URL: https://github.com/apache/atlas/pull/61#discussion_r301804522
 
 

 ##
 File path: distro/src/bin/atlas_start.py
 ##
 @@ -143,13 +135,10 @@ def main():
 web_app_path = os.path.join(web_app_dir, "atlas")
 if (mc.isCygwin()):
 web_app_path = mc.convertCygwinPath(web_app_path)
-if not is_setup:
-start_atlas_server(atlas_classpath, atlas_pid_file, jvm_logdir, 
jvm_opts_list, web_app_path)
-mc.wait_for_startup(confdir, 300)
-print "Apache Atlas Server started!!!\n"
-else:
-process = mc.java("org.apache.atlas.web.setup.AtlasSetup", [], 
atlas_classpath, jvm_opts_list, jvm_logdir)
 
 Review comment:
   To go along with this, the current script fails. And this line needs to be 
changed to:
   ```py
   process = mc.java("org.apache.atlas.web.setup.AtlasSetupModule", [], 
atlas_classpath, jvm_opts_list, jvm_logdir)
   ```
   otherwise, `-setup` fails.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/
---

Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Nixon Rodrigues.


Bugs: ATLAS-3321
https://issues.apache.org/jira/browse/ATLAS-3321


Repository: atlas


Description
---

Atlas has the notion of clusterName which is used as a global namespace in 
which you can define the context in which entity instances are defined and 
grouped. A cluster name defines a scope of operation in atlas and is specified 
in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File 
System), storm_topology (Storm), spark_db (Spark)

cluster name is also used to construct qualifiedName attribute in atlas -

hive_db - db_name@cluster_name
hive_table - db_name.table_name@cluster_name
hive_column - db_name.table_name.column_name@cluster_name
Cluster name works for simple environments wherein we have a single cluster and 
single service instances running (single Hive or HBase service). In complex 
deployments - a single cluster may host multiple instances of the same service 
(two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't 
help to provide context to individual service instances and metadata groupings.

So we introduce - "metadata namespace" where a single cluster can contain 
multiple metadata namespaces. The qualified name for atlas will be generated 
using metadata namespace name instead of cluster name.

For each hook enabled we specify the property - "atlas.metadata.namespace" 
which defines the scope/context of the metadata.


Diffs
-

  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
 1825cd290 
  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
 1765c1848 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 5f8f846ae 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
 76d6fe612 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
ffa56cea0 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 05d79df29 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
cbee7bf14 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
 d55aa5370 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
 1305f65a6 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 b5fdb6dbd 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 4ea484f52 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 f1d02375c 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 8ebb385cf 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
 a7b9b0cd0 
  
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
 8755c9e60 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
5397a4b0b 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 97668a3c6 
  common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 


Diff: https://reviews.apache.org/r/71042/diff/1/


Testing
---

Testing in progress


Thanks,

Sarath Subramanian

Re: Review Request 70913: ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes

[Merryle Wang via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70913/
---

(Updated July 9, 2019, 11:55 p.m.)


Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, and Sarath Subramanian.


Changes
---

The code now takes the hook generated dynAttribute if there isn't a template 
for a given Entity in the models file.


Repository: atlas


Description
---

ATLAS-3286: Populated dynamic attribute flags for AtlasAttributes


Diffs (updated)
-

  addons/models/1000-Hadoop/1030-hive_model.json 
8901aa4aa86fb2802a9e9b1e65c2ff8aad8855ad 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
d9ae9e36773146fad652a1e28fc8822bae5c8557 
  intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 
0fe47bdcbe84c06545c517fec3177ef7e6487a6c 
  intg/src/main/java/org/apache/atlas/type/AttributeToken.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/ConstantToken.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/DependentToken.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/TemplateToken.java PRE-CREATION 
  intg/src/test/java/org/apache/atlas/type/TestAtlasEntityType.java 
3c53c02b48747515217c9327c98209a48ee84237 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java
 4ff4206471bb8b2b4997c00dca3b4433626ae392 


Diff: https://reviews.apache.org/r/70913/diff/10/

Changes: https://reviews.apache.org/r/70913/diff/9-10/


Testing
---

Added unit test


Thanks,

Merryle Wang

Re: Review Request 71006: [ATLAS-3313] Enhance Atlas server to process notifications from multiple Kafka topics

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/#review216452
---




intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 91 (patched)


I agree. Let's rename to getStringArray() - to be in sync with method names 
in  org.apache.commons.configuration.Configuration.



intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
Lines 100 (patched)


I suggest we leave trimming and handling of empty value to the callers. It 
is possible that an empty string is a valid value for some configurations.



intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java
Line 58 (original), 58 (patched)


metrics values are not persisted in Atlas. And given we need to track 
offsets for multiple topics, the metrics name need to change.


- Madhan Neethiraj


On July 8, 2019, 1:26 p.m., Saqeeb Shaikh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71006/
> ---
> 
> (Updated July 8, 2019, 1:26 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
> Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3313
> https://issues.apache.org/jira/browse/ATLAS-3313
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Atlas server processes notifications from Kafka topic named 
> ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). 
> The name of the Kafka topic to which an Atlas Hook publishes notifications 
> can be configured via configuration atlas.notification.hook.topic.name. 
> However, since Atlas server processes notifications from a single topic, all 
> hooks (Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
> 
> To enable different hooks to publish to different Kafka topics, Atlas server 
> should support processing notifications from multiple Kafka topics.
> 
> 
> Add following properties in atlas:
> atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
>  (comma separated without space)
> atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
>   intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> 5c840c3 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> b04aba9 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 449eb6f 
>   notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 
> 847caa3 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
>  263903b 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  c963830 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
> d5a4412 
>   repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java 
> f658caa 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> b56019e 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f8e810 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
>  fb3ff26 
>   
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
>  3e35511 
> 
> 
> Diff: https://reviews.apache.org/r/71006/diff/2/
> 
> 
> Testing
> ---
> 
> Tested by publishing create hive table JSON using kafka-console-producer on 
> custom topics configured above. I was able to search the entities in atlas. 
> Also tested basic use case by creating database and table from hive console. 
> 
> mvn clean package 
> Some test cases are failing on my local (will work on fixing the same).
> 
> 
> Thanks,
> 
> Saqeeb Shaikh
> 
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216460
---




addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 59 (original), 60 (patched)


Configurations "atlas.cluster.name", "atlas.metadata.namespace" and their 
default values - are better better handled in base class AtlasHook, to avoid 
having to deal with this in every hook. Please review and consider refactoring.



addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 60 (original), 61 (patched)


I suggest to use earlier default value - "primary".



addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 109 (original), 112 (patched)


Is it necessary to retain 'clusterName' field?

Please review clusterName in other classes as well, like:
- HBaseBridge
- HiveHook
- ImpalaLineageHook


- Madhan Neethiraj


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216468
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
Line 256 (original), 260 (patched)


Duplicate code.



addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
Line 146 (original), 151 (patched)


Duplicate code.



addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Line 65 (original), 65 (patched)


Can be renamed to be made consistent with other hooks.



addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 176 (patched)


Duplicate code.


- Ashutosh Mestry


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
> 
> 
> Diff: https://reviews.apache.org/r/71042/diff/1/
> 
> 
> Testing
> ---
> 
> Testing in progress
> 
> 
> Thanks,
> 
> Sarath Su

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216469
---



Will make sense to write a unit test that exercises the logic for 
getMetadataNamespace.

- Ashutosh Mestry


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
> 
> 
> Diff: https://reviews.apache.org/r/71042/diff/1/
> 
> 
> Testing
> ---
> 
> Testing in progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

[jira] [Created] (ATLAS-3322) Search Weight used for quick search is not getting applied properly.

[Sridhar (JIRA)]

Sridhar created ATLAS-3322:
--

 Summary: Search Weight used for quick search is not getting 
applied properly.
 Key: ATLAS-3322
 URL: https://issues.apache.org/jira/browse/ATLAS-3322
 Project: Atlas
  Issue Type: Bug
Reporter: Sridhar
Assignee: Sridhar


The search weight patches are not getting applied properly because of schema 
version mis-matches. 





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (ATLAS-3323) Atlas log4j configurations for file-size and file-backup are wrongly placed.

[Vishal Suvagia (JIRA)]

Vishal Suvagia created ATLAS-3323:
-

 Summary: Atlas log4j configurations for file-size and file-backup 
are wrongly placed.
 Key: ATLAS-3323
 URL: https://issues.apache.org/jira/browse/ATLAS-3323
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 2.0.0
Reporter: Vishal Suvagia
Assignee: Vishal Suvagia
 Fix For: 2.1.0


Atlas log4j configurations have maxFileSize and maxBackupIndex configured 
wrongly for File, Audit, Metrics and Failed appenders. Need to move the same in 
the appender section instead of the appender/layout section.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-3253) When I register Atlas Hook in Hbase-site.xml by adding the following:Restart HBase Error

[Vishal Suvagia (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881745#comment-16881745
 ] 

Vishal Suvagia commented on ATLAS-3253:
---

[~langxia], any updates ?

> When I register Atlas Hook in Hbase-site.xml by adding the following:Restart 
> HBase Error
> 
>
> Key: ATLAS-3253
> URL: https://issues.apache.org/jira/browse/ATLAS-3253
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 1.1.0
> Environment: CDH5.7
> atlas version：1.1.0
>Reporter: He sheng
>Priority: Major
>
> | |The coprocessor org.apache.atlas.hbase.hook.HBaseAtlasCoprocessor threw 
> java.lang.ClassNotFoundException: 
> org.apache.atlas.hbase.hook.HBaseAtlasCoprocessor
> java.lang.ClassNotFoundException: 
> org.apache.atlas.hbase.hook.HBaseAtlasCoprocessor
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:158)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.(MasterCoprocessorHost.java:87)
>   at 
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:691)
>   at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:188)
>   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1777)
>   at java.lang.Thread.run(Thread.java:745)|
> The
> !moz-extension://8260c77a-a03d-49f1-8b6d-70c335e701e7/static/drowDown.svg!
> !moz-extension://8260c77a-a03d-49f1-8b6d-70c335e701e7/static/sound.svg!
> 冠词那个
> 定义
> 冠词denoting one or more people or things already mentioned or assumed to be 
> common knowledge.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

[Sridhar K]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
---



Hello Sarath,
Is this some thing new that we are doing? I understand the reasoning. I don't 
see a need for configuration "atlas.metadata.namespace". I suggest that we just 
use the . format. It is hard to manage with 
configurations like this. What would happen if this configuration is changed? 
What will you do with existing objects when this value is changed. This needs 
some kind of design exercise. Can we please have a . meeting for it.

- Sridhar K


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 5397a4b0b 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  97668a3c6 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d 
> 
> 
> Diff: https://reviews.apache.org/r/71042/diff/1/
> 
> 
> Testing
> ---
> 
> Testing in progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

[Madhan Neethiraj]

> On July 10, 2019, 5:55 a.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I 
> > don't see a need for configuration "atlas.metadata.namespace". I suggest 
> > that we just use the . format. It is hard to 
> > manage with configurations like this. What would happen if this 
> > configuration is changed? What will you do with existing objects when this 
> > value is changed. This needs some kind of design exercise. Can we please 
> > have a . meeting for it.

Sridhar - current use of 'clusterName' works when all services that access 
(CRUD) a resource run in the same cluster. There are cases where services 
running in a cluster that is different from where the resource resides. For 
example, consider 2 clusters cl1 and cl2. HiveServer2 services running in both 
clusters can access Hive metastore running in "cl1". In such scenarios holding 
on to 'clusterName' would be confusing; hence the move to a generic name 
'metadata.namespace'. It makes it intutive to ask the services set this 
configuration, instead of confusing them with 'clusterName'. Hope this helps.


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
---


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  1305f65a6 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  b5fdb6dbd 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  4ea484f52 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  f1d02375c 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  8ebb385cf 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java
>  a7b9b0cd0 
>   
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
>  8755c9e60 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hoo

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

[Sridhar K]

> On July 10, 2019, 5:55 a.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I 
> > don't see a need for configuration "atlas.metadata.namespace". I suggest 
> > that we just use the . format. It is hard to 
> > manage with configurations like this. What would happen if this 
> > configuration is changed? What will you do with existing objects when this 
> > value is changed. This needs some kind of design exercise. Can we please 
> > have a . meeting for it.
> 
> Madhan Neethiraj wrote:
> Sridhar - current use of 'clusterName' works when all services that 
> access (CRUD) a resource run in the same cluster. There are cases where 
> services running in a cluster that is different from where the resource 
> resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services 
> running in both clusters can access Hive metastore running in "cl1". In such 
> scenarios holding on to 'clusterName' would be confusing; hence the move to a 
> generic name 'metadata.namespace'. It makes it intutive to ask the services 
> set this configuration, instead of confusing them with 'clusterName'. Hope 
> this helps.

I see your point. But, the issue is in the possibility of wrongly configured 
names and flexibility of changing the names. In the example you gave, I would 
expect HMS to have the convention of "cl1.hms1" as auto generated configuration 
and Hive-Server resources like hive_processes should have their name spaces as 
"cl1.hive-server-1" and " cl2.hive-server-1". I think that this gives a general 
understanding on namespace hierarchy. If it is user configurable, user might 
accidentally give the same name for every cluster resulting in mixups. So, I 
think that we need to give some thinking. You might have given some thinking on 
it. If so,  I would like to understand it.


- Sridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
---


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
> cbee7bf14 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
>  d55aa5370 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookConte

Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace

[Sridhar K]

> On July 10, 2019, 5:55 a.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I 
> > don't see a need for configuration "atlas.metadata.namespace". I suggest 
> > that we just use the . format. It is hard to 
> > manage with configurations like this. What would happen if this 
> > configuration is changed? What will you do with existing objects when this 
> > value is changed. This needs some kind of design exercise. Can we please 
> > have a . meeting for it.
> 
> Madhan Neethiraj wrote:
> Sridhar - current use of 'clusterName' works when all services that 
> access (CRUD) a resource run in the same cluster. There are cases where 
> services running in a cluster that is different from where the resource 
> resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services 
> running in both clusters can access Hive metastore running in "cl1". In such 
> scenarios holding on to 'clusterName' would be confusing; hence the move to a 
> generic name 'metadata.namespace'. It makes it intutive to ask the services 
> set this configuration, instead of confusing them with 'clusterName'. Hope 
> this helps.
> 
> Sridhar K wrote:
> I see your point. But, the issue is in the possibility of wrongly 
> configured names and flexibility of changing the names. In the example you 
> gave, I would expect HMS to have the convention of "cl1.hms1" as auto 
> generated configuration and Hive-Server resources like hive_processes should 
> have their name spaces as "cl1.hive-server-1" and " cl2.hive-server-1". I 
> think that this gives a general understanding on namespace hierarchy. If it 
> is user configurable, user might accidentally give the same name for every 
> cluster resulting in mixups. So, I think that we need to give some thinking. 
> You might have given some thinking on it. If so,  I would like to understand 
> it.

If we allow the user to give values for namespace, we might have to have this 
information captured as first class Atlas Entity as well--for auditing, 
goveranance point of view.


- Sridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
---


On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> ---
> 
> (Updated July 9, 2019, 11:53 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
> Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Atlas has the notion of clusterName which is used as a global namespace in 
> which you can define the context in which entity instances are defined and 
> grouped. A cluster name defines a scope of operation in atlas and is 
> specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), 
> fs_path (File System), storm_topology (Storm), spark_db (Spark)
> 
> cluster name is also used to construct qualifiedName attribute in atlas -
> 
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster 
> and single service instances running (single Hive or HBase service). In 
> complex deployments - a single cluster may host multiple instances of the 
> same service (two Hive services - Hive-1 and Hive-2), for such cases cluster 
> name doesn't help to provide context to individual service instances and 
> metadata groupings.
> 
> So we introduce - "metadata namespace" where a single cluster can contain 
> multiple metadata namespaces. The qualified name for atlas will be generated 
> using metadata namespace name instead of cluster name.
> 
> For each hook enabled we specify the property - "atlas.metadata.namespace" 
> which defines the scope/context of the metadata.
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  1825cd290 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  1765c1848 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  5f8f846ae 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  76d6fe612 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffa56cea0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  05d79df29 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITB