[atlas] branch branch-2.0 updated: ATLAS-4458: Commons logging reference fix.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 0d41588 ATLAS-4458: Commons logging reference fix.
0d41588 is described below
commit 0d41588ea156f6ce45b83f7efc52a85bf15d1038
Author: Ashutosh Mestry
AuthorDate: Wed Oct 20 21:26:13 2021 -0700
ATLAS-4458: Commons logging reference fix.
---
intg/pom.xml | 10 --
notification/pom.xml | 2 +-
repository/pom.xml | 10 +-
webapp/pom.xml | 32
4 files changed, 2 insertions(+), 52 deletions(-)
diff --git a/intg/pom.xml b/intg/pom.xml
index f4391c1..ffccbca 100644
--- a/intg/pom.xml
+++ b/intg/pom.xml
@@ -55,10 +55,6 @@
io.netty
netty-transport-native-epoll
-
-commons-logging
-commons-logging
-
@@ -66,12 +62,6 @@
commons-validator
commons-validator
${commons-validator.version}
-
-
-commons-logging
-commons-logging
-
-
diff --git a/notification/pom.xml b/notification/pom.xml
index 26b7443..e8abd45 100644
--- a/notification/pom.xml
+++ b/notification/pom.xml
@@ -88,7 +88,7 @@
tests
test
-
+
diff --git a/repository/pom.xml b/repository/pom.xml
index ee276d5..0c19ad6 100755
--- a/repository/pom.xml
+++ b/repository/pom.xml
@@ -41,10 +41,6 @@
javax.servlet
servlet-api
-
-commons-logging
-commons-logging
-
@@ -191,11 +187,7 @@
org.mortbay.jetty
servlet-api-2.5
-
-
-commons-logging
-commons-logging
-
+
diff --git a/webapp/pom.xml b/webapp/pom.xml
index ddbd8e7..b5d52ba 100755
--- a/webapp/pom.xml
+++ b/webapp/pom.xml
@@ -149,12 +149,6 @@
org.apache.atlas
atlas-intg
-
-
-commons-logging
-commons-logging
-
-
@@ -225,10 +219,6 @@
io.netty
netty-transport-native-epoll
-
-commons-logging
-commons-logging
-
@@ -266,12 +256,6 @@
org.apache.httpcomponents
httpclient
-
-
-commons-logging
-commons-logging
-
-
@@ -282,12 +266,6 @@
commons-configuration
commons-configuration
-
-
-commons-logging
-commons-logging
-
-
@@ -400,12 +378,6 @@
org.springframework
spring-core
-
-
-commons-logging
-commons-logging
-
-
@@ -520,10 +492,6 @@
org.elasticsearch
elasticsearch
-
-commons-logging
-commons-logging
-
[atlas] branch master updated: ATLAS-4458: Commons logging reference fix.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new fc60a07 ATLAS-4458: Commons logging reference fix.
fc60a07 is described below
commit fc60a07e4de4fcd7a4fc5d6659dc860f4f55b4fa
Author: Ashutosh Mestry
AuthorDate: Wed Oct 20 21:26:13 2021 -0700
ATLAS-4458: Commons logging reference fix.
---
intg/pom.xml | 10 --
notification/pom.xml | 2 +-
repository/pom.xml | 10 +-
webapp/pom.xml | 32
4 files changed, 2 insertions(+), 52 deletions(-)
diff --git a/intg/pom.xml b/intg/pom.xml
index 691cdf0..40031cb 100644
--- a/intg/pom.xml
+++ b/intg/pom.xml
@@ -55,10 +55,6 @@
io.netty
netty-transport-native-epoll
-
-commons-logging
-commons-logging
-
@@ -66,12 +62,6 @@
commons-validator
commons-validator
${commons-validator.version}
-
-
-commons-logging
-commons-logging
-
-
diff --git a/notification/pom.xml b/notification/pom.xml
index e4ef3b3..aaf11c7 100644
--- a/notification/pom.xml
+++ b/notification/pom.xml
@@ -88,7 +88,7 @@
tests
test
-
+
diff --git a/repository/pom.xml b/repository/pom.xml
index 4cb45b2..d9488af 100755
--- a/repository/pom.xml
+++ b/repository/pom.xml
@@ -41,10 +41,6 @@
javax.servlet
servlet-api
-
-commons-logging
-commons-logging
-
@@ -191,11 +187,7 @@
org.mortbay.jetty
servlet-api-2.5
-
-
-commons-logging
-commons-logging
-
+
diff --git a/webapp/pom.xml b/webapp/pom.xml
index 0e4e21d..443559c 100755
--- a/webapp/pom.xml
+++ b/webapp/pom.xml
@@ -149,12 +149,6 @@
org.apache.atlas
atlas-intg
-
-
-commons-logging
-commons-logging
-
-
@@ -225,10 +219,6 @@
io.netty
netty-transport-native-epoll
-
-commons-logging
-commons-logging
-
@@ -266,12 +256,6 @@
org.apache.httpcomponents
httpclient
-
-
-commons-logging
-commons-logging
-
-
@@ -282,12 +266,6 @@
commons-configuration
commons-configuration
-
-
-commons-logging
-commons-logging
-
-
@@ -400,12 +378,6 @@
org.springframework
spring-core
-
-
-commons-logging
-commons-logging
-
-
@@ -520,10 +492,6 @@
org.elasticsearch
elasticsearch
-
-commons-logging
-commons-logging
-
[atlas] branch branch-2.0 updated: ATLAS-4444: Atlas - Upgrade commons-io to 2.11.0
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new 0004b69 ATLAS-: Atlas - Upgrade commons-io to 2.11.0 0004b69 is described below commit 0004b69ed4ffae75f29bb444efdab2012161e303 Author: Disha Talreja AuthorDate: Tue Oct 5 11:48:49 2021 -0400 ATLAS-: Atlas - Upgrade commons-io to 2.11.0 Signed-off-by: Ashutosh Mestry --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 49643a1..023de75 100644 --- a/pom.xml +++ b/pom.xml @@ -690,7 +690,7 @@ 1.10 2.2 1.0 -2.8.0 +2.11.0 2.6 1.1.3 1.6
[atlas] branch master updated: ATLAS-4444: Atlas - Upgrade commons-io to 2.11.0
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git The following commit(s) were added to refs/heads/master by this push: new 375a6e4 ATLAS-: Atlas - Upgrade commons-io to 2.11.0 375a6e4 is described below commit 375a6e43f1cf3241172a7955028e11012f2c4dea Author: Disha Talreja AuthorDate: Tue Oct 5 11:48:49 2021 -0400 ATLAS-: Atlas - Upgrade commons-io to 2.11.0 Signed-off-by: Ashutosh Mestry --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 7ccf0af..84630e7 100644 --- a/pom.xml +++ b/pom.xml @@ -690,7 +690,7 @@ 1.10 2.2 1.0 -2.8.0 +2.11.0 2.6 1.1.3 1.6
[atlas] branch branch-2.0 updated: ATLAS-4440: Update Kafka dependency to 2.8.1
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 3007b44 ATLAS-4440: Update Kafka dependency to 2.8.1
3007b44 is described below
commit 3007b4475eb79b71c6505d801a191540be63c1e3
Author: Viktor Somogyi-Vass
AuthorDate: Thu Sep 30 10:58:11 2021 +0200
ATLAS-4440: Update Kafka dependency to 2.8.1
Signed-off-by: Ashutosh Mestry
---
.../src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java | 2 +-
pom.xml | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
b/notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
index b793b9a..19717fb 100644
--- a/notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
+++ b/notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
@@ -136,7 +136,7 @@ public class EmbeddedKafkaServer implements Service {
brokerConfig.setProperty("log.dirs",
constructDir("kafka").getAbsolutePath());
brokerConfig.setProperty("log.flush.interval.messages",
String.valueOf(1));
-kafkaServer = new KafkaServer(KafkaConfig.fromProps(brokerConfig),
Time.SYSTEM, Option.apply(this.getClass().getName()), new ArrayBuffer<>());
+kafkaServer = new KafkaServer(KafkaConfig.fromProps(brokerConfig),
Time.SYSTEM, Option.apply(this.getClass().getName()), false);
kafkaServer.startup();
diff --git a/pom.xml b/pom.xml
index 7642ad6..49643a1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -730,7 +730,7 @@
1.1
4.13.1
2.12
-2.5.0
+2.8.1
6.0.1
1.2.17
2.13.3
[atlas] branch master updated: ATLAS-4440: Update Kafka dependency to 2.8.1
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 028e3e0 ATLAS-4440: Update Kafka dependency to 2.8.1
028e3e0 is described below
commit 028e3e0b85ee63619924a157f1bf790a2fd9f213
Author: Viktor Somogyi-Vass
AuthorDate: Thu Sep 30 10:58:11 2021 +0200
ATLAS-4440: Update Kafka dependency to 2.8.1
Signed-off-by: Ashutosh Mestry
---
.../src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java | 2 +-
pom.xml | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
b/notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
index b793b9a..19717fb 100644
--- a/notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
+++ b/notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java
@@ -136,7 +136,7 @@ public class EmbeddedKafkaServer implements Service {
brokerConfig.setProperty("log.dirs",
constructDir("kafka").getAbsolutePath());
brokerConfig.setProperty("log.flush.interval.messages",
String.valueOf(1));
-kafkaServer = new KafkaServer(KafkaConfig.fromProps(brokerConfig),
Time.SYSTEM, Option.apply(this.getClass().getName()), new ArrayBuffer<>());
+kafkaServer = new KafkaServer(KafkaConfig.fromProps(brokerConfig),
Time.SYSTEM, Option.apply(this.getClass().getName()), false);
kafkaServer.startup();
diff --git a/pom.xml b/pom.xml
index 47768e9..7ccf0af 100644
--- a/pom.xml
+++ b/pom.xml
@@ -730,7 +730,7 @@
1.1
4.13.1
2.12
-2.5.0
+2.8.1
6.0.1
1.2.17
2.13.3
[atlas] branch branch-2.0 updated: ATLAS-4358: Address race condition in AtlasPatchRegistry in HA mode.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new bdadd60 ATLAS-4358: Address race condition in AtlasPatchRegistry in
HA mode.
bdadd60 is described below
commit bdadd60a0f7dfa198b9f1b6f077a6b5301694fc8
Author: Ashutosh Mestry
AuthorDate: Fri Sep 24 10:53:48 2021 -0700
ATLAS-4358: Address race condition in AtlasPatchRegistry in HA mode.
---
.../repository/patches/AtlasPatchManager.java | 45 +-
1 file changed, 27 insertions(+), 18 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
b/repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
index e2a38ab..d30971a 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
@@ -40,28 +40,19 @@ import static
org.apache.atlas.model.patches.AtlasPatch.PatchStatus.SKIPPED;
public class AtlasPatchManager {
private static final Logger LOG =
LoggerFactory.getLogger(AtlasPatchManager.class);
-private final PatchContextcontext;
private final List handlers = new ArrayList<>();
+private final AtlasGraph atlasGraph;
+private final AtlasTypeRegistry typeRegistry;
+private final GraphBackedSearchIndexer indexer;
+private final EntityGraphMapper entityGraphMapper;
+private PatchContextcontext;
@Inject
public AtlasPatchManager(AtlasGraph atlasGraph, AtlasTypeRegistry
typeRegistry, GraphBackedSearchIndexer indexer, EntityGraphMapper
entityGraphMapper) {
-this.context = new PatchContext(atlasGraph, typeRegistry, indexer,
entityGraphMapper);
-}
-
-@PostConstruct
-public void init() {
-LOG.info("==> AtlasPatchManager.init()");
-
-// register all java patches here
-handlers.add(new UniqueAttributePatch(context));
-handlers.add(new ClassificationTextPatch(context));
-handlers.add(new FreeTextRequestHandlerPatch(context));
-handlers.add(new SuggestionsRequestHandlerPatch(context));
-handlers.add(new IndexConsistencyPatch(context));
-handlers.add(new ReIndexPatch(context));
-handlers.add(new ProcessNamePatch(context));
-
-LOG.info("<== AtlasPatchManager.init()");
+this.atlasGraph = atlasGraph;
+this.typeRegistry = typeRegistry;
+this.indexer = indexer;
+this.entityGraphMapper = entityGraphMapper;
}
public AtlasPatches getAllPatches() {
@@ -70,6 +61,7 @@ public class AtlasPatchManager {
public void applyAll() {
LOG.info("==> AtlasPatchManager.applyAll()");
+init();
try {
for (AtlasPatchHandler handler : handlers) {
@@ -90,6 +82,23 @@ public class AtlasPatchManager {
LOG.info("<== AtlasPatchManager.applyAll()");
}
+private void init() {
+LOG.info("==> AtlasPatchManager.init()");
+
+this.context = new PatchContext(atlasGraph, typeRegistry, indexer,
entityGraphMapper);
+
+// register all java patches here
+handlers.add(new UniqueAttributePatch(context));
+handlers.add(new ClassificationTextPatch(context));
+handlers.add(new FreeTextRequestHandlerPatch(context));
+handlers.add(new SuggestionsRequestHandlerPatch(context));
+handlers.add(new IndexConsistencyPatch(context));
+handlers.add(new ReIndexPatch(context));
+handlers.add(new ProcessNamePatch(context));
+
+LOG.info("<== AtlasPatchManager.init()");
+}
+
public void addPatchHandler(AtlasPatchHandler patchHandler) {
handlers.add(patchHandler);
}
[atlas] branch master updated: ATLAS-4358: Address race condition in AtlasPatchRegistry in HA mode.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new a2eaca4 ATLAS-4358: Address race condition in AtlasPatchRegistry in
HA mode.
a2eaca4 is described below
commit a2eaca4ed9a992b42bed18ba73c1da6ed414566e
Author: Ashutosh Mestry
AuthorDate: Fri Sep 24 10:53:48 2021 -0700
ATLAS-4358: Address race condition in AtlasPatchRegistry in HA mode.
---
.../repository/patches/AtlasPatchManager.java | 45 +-
1 file changed, 27 insertions(+), 18 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
b/repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
index e2a38ab..d30971a 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
@@ -40,28 +40,19 @@ import static
org.apache.atlas.model.patches.AtlasPatch.PatchStatus.SKIPPED;
public class AtlasPatchManager {
private static final Logger LOG =
LoggerFactory.getLogger(AtlasPatchManager.class);
-private final PatchContextcontext;
private final List handlers = new ArrayList<>();
+private final AtlasGraph atlasGraph;
+private final AtlasTypeRegistry typeRegistry;
+private final GraphBackedSearchIndexer indexer;
+private final EntityGraphMapper entityGraphMapper;
+private PatchContextcontext;
@Inject
public AtlasPatchManager(AtlasGraph atlasGraph, AtlasTypeRegistry
typeRegistry, GraphBackedSearchIndexer indexer, EntityGraphMapper
entityGraphMapper) {
-this.context = new PatchContext(atlasGraph, typeRegistry, indexer,
entityGraphMapper);
-}
-
-@PostConstruct
-public void init() {
-LOG.info("==> AtlasPatchManager.init()");
-
-// register all java patches here
-handlers.add(new UniqueAttributePatch(context));
-handlers.add(new ClassificationTextPatch(context));
-handlers.add(new FreeTextRequestHandlerPatch(context));
-handlers.add(new SuggestionsRequestHandlerPatch(context));
-handlers.add(new IndexConsistencyPatch(context));
-handlers.add(new ReIndexPatch(context));
-handlers.add(new ProcessNamePatch(context));
-
-LOG.info("<== AtlasPatchManager.init()");
+this.atlasGraph = atlasGraph;
+this.typeRegistry = typeRegistry;
+this.indexer = indexer;
+this.entityGraphMapper = entityGraphMapper;
}
public AtlasPatches getAllPatches() {
@@ -70,6 +61,7 @@ public class AtlasPatchManager {
public void applyAll() {
LOG.info("==> AtlasPatchManager.applyAll()");
+init();
try {
for (AtlasPatchHandler handler : handlers) {
@@ -90,6 +82,23 @@ public class AtlasPatchManager {
LOG.info("<== AtlasPatchManager.applyAll()");
}
+private void init() {
+LOG.info("==> AtlasPatchManager.init()");
+
+this.context = new PatchContext(atlasGraph, typeRegistry, indexer,
entityGraphMapper);
+
+// register all java patches here
+handlers.add(new UniqueAttributePatch(context));
+handlers.add(new ClassificationTextPatch(context));
+handlers.add(new FreeTextRequestHandlerPatch(context));
+handlers.add(new SuggestionsRequestHandlerPatch(context));
+handlers.add(new IndexConsistencyPatch(context));
+handlers.add(new ReIndexPatch(context));
+handlers.add(new ProcessNamePatch(context));
+
+LOG.info("<== AtlasPatchManager.init()");
+}
+
public void addPatchHandler(AtlasPatchHandler patchHandler) {
handlers.add(patchHandler);
}
[atlas] 01/02: ATLAS-4379 : Atlas Filter changes for user inactivity on Atlas UI
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit be5d557a0038536733730ca47ff473d876415c3f
Author: nixonrodrigues
AuthorDate: Mon Aug 23 17:27:18 2021 +0530
ATLAS-4379 : Atlas Filter changes for user inactivity on Atlas UI
---
.../java/org/apache/atlas/AtlasConfiguration.java | 4 +-
.../web/filters/AtlasAuthenticationFilter.java | 48 -
.../org/apache/atlas/web/filters/RestUtil.java | 108 +
.../apache/atlas/web/resources/AdminResource.java | 1 +
.../AtlasAuthenticationSuccessHandler.java | 16 ++-
5 files changed, 170 insertions(+), 7 deletions(-)
diff --git a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
index 2f2c8a5..fa519ef 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
@@ -80,7 +80,9 @@ public enum AtlasConfiguration {
DSL_EXECUTOR_TRAVERSAL("atlas.dsl.executor.traversal", true),
DSL_CACHED_TRANSLATOR("atlas.dsl.cached.translator", true),
DEBUG_METRICS_ENABLED("atlas.debug.metrics.enabled", false),
-TASKS_USE_ENABLED("atlas.tasks.enabled", true);
+TASKS_USE_ENABLED("atlas.tasks.enabled", true),
+SESSION_TIMEOUT_SECS("atlas.session.timeout.secs", 3600);
+
private static final Configuration APPLICATION_PROPERTIES;
diff --git
a/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
b/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
index d9b1c82..b8d21b9 100644
---
a/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
+++
b/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
@@ -19,6 +19,7 @@
package org.apache.atlas.web.filters;
import org.apache.atlas.ApplicationProperties;
+import org.apache.atlas.AtlasConfiguration;
import org.apache.atlas.security.SecurityProperties;
import org.apache.atlas.utils.AuthenticationUtil;
import org.apache.atlas.web.security.AtlasAuthenticationProvider;
@@ -73,6 +74,9 @@ import java.util.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.hadoop.security.authorize.AuthorizationException;
+import
org.springframework.security.web.authentication.logout.SecurityContextLogoutHandler;
+
+import static org.apache.atlas.web.filters.RestUtil.constructForwardableURL;
/**
@@ -90,6 +94,7 @@ public class AtlasAuthenticationFilter extends
AuthenticationFilter {
private static final String[] DEFAULT_PROXY_USERS = new String[]
{ "knox" };
private static final String CONF_PROXYUSER_PREFIX =
"atlas.proxyuser";
protected static final ServletContext nullContext = new
NullServletContext();
+private static final String ORIGINAL_URL_QUERY_PARAM =
"originalUrl";
private Signer signer;
private SignerSecretProvider secretProvider;
@@ -102,8 +107,9 @@ public class AtlasAuthenticationFilter extends
AuthenticationFilter {
private Set atlasProxyUsers = new HashSet<>();
private HttpServlet optionsServlet;
private boolean supportTrustedProxy = false;
+private int sessionTimeout;
-
+private SecurityContextLogoutHandler logoutHandler;
public AtlasAuthenticationFilter() {
LOG.info("==> AtlasAuthenticationFilter()");
@@ -193,7 +199,7 @@ public class AtlasAuthenticationFilter extends
AuthenticationFilter {
optionsServlet = new HttpServlet() {
};
optionsServlet.init();
-
+logoutHandler = new SecurityContextLogoutHandler();
LOG.info("<== AtlasAuthenticationFilter.init(filterConfig={})",
filterConfig);
}
@@ -301,6 +307,10 @@ public class AtlasAuthenticationFilter extends
AuthenticationFilter {
LOG.debug(" AuthenticationFilterConfig: {}", ret);
+sessionTimeout = AtlasConfiguration.SESSION_TIMEOUT_SECS.getInt();
+if(sessionTimeout < 30){
+LOG.warn("AtlasAuthenticationFilter:: sessionTimeout is set low");
+}
supportKeyTabBrowserLogin =
configuration.getBoolean("atlas.authentication.method.kerberos.support.keytab.browser.login",
false);
supportTrustedProxy =
configuration.getBoolean("atlas.authentication.method.trustedproxy", true);
String agents =
configuration.getString(AtlasCSRFPreventionFilter.BROWSER_USER_AGENT_PARAM,
AtlasCSRFPreventionFilter.BROWSER_USER_AGENTS_DEFAULT);
@@ -332,6 +342,8 @@ public class AtlasAuthenticationFilter extends
AuthenticationFilter {
[atlas] 02/02: ATLAS-4435: Disable session inactivity timeout.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 56ae53b37533de32573e3a9314667525311da600
Author: Ashutosh Mestry
AuthorDate: Wed Sep 22 12:31:57 2021 -0700
ATLAS-4435: Disable session inactivity timeout.
---
.../java/org/apache/atlas/AtlasConfiguration.java | 2 +-
.../web/filters/AtlasAuthenticationFilter.java | 41 --
.../apache/atlas/web/resources/AdminResource.java | 5 ++-
.../AtlasAuthenticationSuccessHandler.java | 5 ++-
4 files changed, 39 insertions(+), 14 deletions(-)
diff --git a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
index fa519ef..20f8f73 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
@@ -81,7 +81,7 @@ public enum AtlasConfiguration {
DSL_CACHED_TRANSLATOR("atlas.dsl.cached.translator", true),
DEBUG_METRICS_ENABLED("atlas.debug.metrics.enabled", false),
TASKS_USE_ENABLED("atlas.tasks.enabled", true),
-SESSION_TIMEOUT_SECS("atlas.session.timeout.secs", 3600);
+SESSION_TIMEOUT_SECS("atlas.session.timeout.secs", -1);
private static final Configuration APPLICATION_PROPERTIES;
diff --git
a/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
b/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
index b8d21b9..6ad0da1 100644
---
a/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
+++
b/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
@@ -29,6 +29,7 @@ import org.apache.commons.configuration.Configuration;
import org.apache.commons.configuration.ConfigurationConverter;
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.security.authentication.client.AuthenticatedURL;
import
org.apache.hadoop.security.authentication.client.AuthenticationException;
import org.apache.hadoop.security.authentication.client.KerberosAuthenticator;
@@ -39,6 +40,7 @@ import
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHa
import org.apache.hadoop.security.authentication.util.Signer;
import org.apache.hadoop.security.authentication.util.SignerException;
import org.apache.hadoop.security.authentication.util.SignerSecretProvider;
+import org.apache.hadoop.security.authorize.AuthorizationException;
import org.apache.hadoop.security.authorize.ProxyUsers;
import org.apache.log4j.NDC;
import org.slf4j.Logger;
@@ -51,8 +53,9 @@ import
org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.core.userdetails.User;
import org.springframework.security.core.userdetails.UserDetails;
import
org.springframework.security.web.authentication.WebAuthenticationDetails;
+import
org.springframework.security.web.authentication.logout.SecurityContextLogoutHandler;
import org.springframework.stereotype.Component;
-import org.apache.hadoop.security.UserGroupInformation;
+
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletContext;
@@ -70,11 +73,19 @@ import java.net.InetAddress;
import java.net.UnknownHostException;
import java.security.Principal;
import java.text.SimpleDateFormat;
-import java.util.*;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Date;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.TimeZone;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import org.apache.hadoop.security.authorize.AuthorizationException;
-import
org.springframework.security.web.authentication.logout.SecurityContextLogoutHandler;
import static org.apache.atlas.web.filters.RestUtil.constructForwardableURL;
@@ -88,6 +99,7 @@ import static
org.apache.atlas.web.filters.RestUtil.constructForwardableURL;
public class AtlasAuthenticationFilter extends AuthenticationFilter {
private static final Logger LOG =
LoggerFactory.getLogger(AtlasAuthenticationFilter.class);
+private static final intSESSION_TIMEOUT_DISABLED_VALUE = -1;
private static final String CONFIG_KERBEROS_TOKEN_VALIDITY =
"atlas.authentication.method.kerberos.token.validity";
private static final String CONFIG_PROXY_USERS=
"atlas.proxyusers";
private static final String PREFIX=
"atlas.authentication.method";
@@ -199,7 +211,11 @@ public class AtlasAuthentication
[atlas] branch branch-2.0 updated (13f1d81 -> 56ae53b)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git. from 13f1d81 ATLAS-4426:UI:#1 In search results page, user should not be able to access the option for applying term or classification for historical entity, fixed new be5d557 ATLAS-4379 : Atlas Filter changes for user inactivity on Atlas UI new 56ae53b ATLAS-4435: Disable session inactivity timeout. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/atlas/AtlasConfiguration.java | 4 +- .../web/filters/AtlasAuthenticationFilter.java | 69 - .../org/apache/atlas/web/filters/RestUtil.java | 108 + .../apache/atlas/web/resources/AdminResource.java | 4 + .../AtlasAuthenticationSuccessHandler.java | 19 +++- 5 files changed, 196 insertions(+), 8 deletions(-) create mode 100644 webapp/src/main/java/org/apache/atlas/web/filters/RestUtil.java
[atlas] branch master updated: ATLAS-4435: Disable session inactivity timeout.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 909d953 ATLAS-4435: Disable session inactivity timeout.
909d953 is described below
commit 909d9531ca8031144f82e2be963dc27b5e780b4e
Author: Ashutosh Mestry
AuthorDate: Wed Sep 22 12:31:57 2021 -0700
ATLAS-4435: Disable session inactivity timeout.
---
.../java/org/apache/atlas/AtlasConfiguration.java | 2 +-
.../web/filters/AtlasAuthenticationFilter.java | 41 --
.../apache/atlas/web/resources/AdminResource.java | 5 ++-
.../AtlasAuthenticationSuccessHandler.java | 5 ++-
4 files changed, 39 insertions(+), 14 deletions(-)
diff --git a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
index fa519ef..20f8f73 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
@@ -81,7 +81,7 @@ public enum AtlasConfiguration {
DSL_CACHED_TRANSLATOR("atlas.dsl.cached.translator", true),
DEBUG_METRICS_ENABLED("atlas.debug.metrics.enabled", false),
TASKS_USE_ENABLED("atlas.tasks.enabled", true),
-SESSION_TIMEOUT_SECS("atlas.session.timeout.secs", 3600);
+SESSION_TIMEOUT_SECS("atlas.session.timeout.secs", -1);
private static final Configuration APPLICATION_PROPERTIES;
diff --git
a/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
b/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
index b8d21b9..6ad0da1 100644
---
a/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
+++
b/webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
@@ -29,6 +29,7 @@ import org.apache.commons.configuration.Configuration;
import org.apache.commons.configuration.ConfigurationConverter;
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.security.authentication.client.AuthenticatedURL;
import
org.apache.hadoop.security.authentication.client.AuthenticationException;
import org.apache.hadoop.security.authentication.client.KerberosAuthenticator;
@@ -39,6 +40,7 @@ import
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHa
import org.apache.hadoop.security.authentication.util.Signer;
import org.apache.hadoop.security.authentication.util.SignerException;
import org.apache.hadoop.security.authentication.util.SignerSecretProvider;
+import org.apache.hadoop.security.authorize.AuthorizationException;
import org.apache.hadoop.security.authorize.ProxyUsers;
import org.apache.log4j.NDC;
import org.slf4j.Logger;
@@ -51,8 +53,9 @@ import
org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.core.userdetails.User;
import org.springframework.security.core.userdetails.UserDetails;
import
org.springframework.security.web.authentication.WebAuthenticationDetails;
+import
org.springframework.security.web.authentication.logout.SecurityContextLogoutHandler;
import org.springframework.stereotype.Component;
-import org.apache.hadoop.security.UserGroupInformation;
+
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletContext;
@@ -70,11 +73,19 @@ import java.net.InetAddress;
import java.net.UnknownHostException;
import java.security.Principal;
import java.text.SimpleDateFormat;
-import java.util.*;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Date;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.TimeZone;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import org.apache.hadoop.security.authorize.AuthorizationException;
-import
org.springframework.security.web.authentication.logout.SecurityContextLogoutHandler;
import static org.apache.atlas.web.filters.RestUtil.constructForwardableURL;
@@ -88,6 +99,7 @@ import static
org.apache.atlas.web.filters.RestUtil.constructForwardableURL;
public class AtlasAuthenticationFilter extends AuthenticationFilter {
private static final Logger LOG =
LoggerFactory.getLogger(AtlasAuthenticationFilter.class);
+private static final intSESSION_TIMEOUT_DISABLED_VALUE = -1;
private static final String CONFIG_KERBEROS_TOKEN_VALIDITY =
"atlas.authentication.method.kerberos.token.validity";
private static final String CONFIG_PROXY_USERS=
"atl
[atlas] branch branch-2.0 updated (1a2ca74 -> f202be6)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git. from 1a2ca74 ATLAS-4403: Improve find entity by unique attributes logic - when multiple unique attributes exist for entity type new c1a3657 ATLAS-4402: Upgrade junit to 4.13.1 new 34bdfed ATLAS-4401: Upgrade commons-io to 2.8.0 new f202be6 ATLAS-4406: Ignore configs:- A hive table DDL entity is created for ignored hive tables The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pom.xml | 4 ++-- .../org/apache/atlas/notification/preprocessor/HivePreprocessor.java | 1 + 2 files changed, 3 insertions(+), 2 deletions(-)
[atlas] 02/03: ATLAS-4401: Upgrade commons-io to 2.8.0
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git commit 34bdfed26bbf2655c49348fd2e45e13a8f88012e Author: Disha Talreja AuthorDate: Wed Aug 25 11:27:46 2021 -0400 ATLAS-4401: Upgrade commons-io to 2.8.0 --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 7b8afef..a2797b3 100644 --- a/pom.xml +++ b/pom.xml @@ -690,7 +690,7 @@ 1.10 2.2 1.0 -2.6 +2.8.0 2.6 1.1.3 1.6
[atlas] 03/03: ATLAS-4406: Ignore configs:- A hive table DDL entity is created for ignored hive tables
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit f202be6e5d14cfbd4f2e5d8607fbb43be25a10dd
Author: Disha Talreja
AuthorDate: Thu Aug 26 22:10:08 2021 -0400
ATLAS-4406: Ignore configs:- A hive table DDL entity is created for ignored
hive tables
---
.../org/apache/atlas/notification/preprocessor/HivePreprocessor.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
b/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
index bf6a623..083e343 100644
---
a/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
+++
b/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
@@ -71,6 +71,7 @@ public class HivePreprocessor {
context.addToIgnoredEntities(entity.getAttribute(ATTRIBUTE_SD));
context.addToIgnoredEntities(entity.getAttribute(ATTRIBUTE_COLUMNS));
context.addToIgnoredEntities(entity.getAttribute(ATTRIBUTE_PARTITION_KEYS));
+
context.addToIgnoredEntities(entity.getAttribute(TYPE_HIVE_TABLE_DDL));
} else if (action == PreprocessAction.PRUNE) {
context.addToPrunedEntities(entity);
[atlas] 01/03: ATLAS-4402: Upgrade junit to 4.13.1
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git commit c1a3657f5e9611087f9271971bb8d9d7996d6346 Author: Disha Talreja AuthorDate: Wed Aug 25 14:57:02 2021 -0400 ATLAS-4402: Upgrade junit to 4.13.1 --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index af55c72..7b8afef 100644 --- a/pom.xml +++ b/pom.xml @@ -728,7 +728,7 @@ 2.10.6 3.2.11 1.1 -4.13 +4.13.1 2.12 2.5.0 6.0.1
[atlas] 02/03: ATLAS-4401: Upgrade commons-io to 2.8.0
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git commit 38caee17c9a80ea6196ec3363e0f3d58c905c580 Author: Disha Talreja AuthorDate: Wed Aug 25 11:27:46 2021 -0400 ATLAS-4401: Upgrade commons-io to 2.8.0 --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 7290db1..263b3e1 100644 --- a/pom.xml +++ b/pom.xml @@ -690,7 +690,7 @@ 1.10 2.2 1.0 -2.6 +2.8.0 2.6 1.1.3 1.6
[atlas] branch master updated (5deac62 -> e962143)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git. from 5deac62 ATLAS-4378: UI - Implement session timeout on Atlas UI. new 50203e0 ATLAS-4402: Upgrade junit to 4.13.1 new 38caee1 ATLAS-4401: Upgrade commons-io to 2.8.0 new e962143 ATLAS-4406: Ignore configs:- A hive table DDL entity is created for ignored hive tables The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pom.xml | 4 ++-- .../org/apache/atlas/notification/preprocessor/HivePreprocessor.java | 1 + 2 files changed, 3 insertions(+), 2 deletions(-)
[atlas] 01/03: ATLAS-4402: Upgrade junit to 4.13.1
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git commit 50203e025852eda970bc812d9ba5ed2e5ba4242b Author: Disha Talreja AuthorDate: Wed Aug 25 14:57:02 2021 -0400 ATLAS-4402: Upgrade junit to 4.13.1 --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 8ae338c..7290db1 100644 --- a/pom.xml +++ b/pom.xml @@ -728,7 +728,7 @@ 2.10.6 3.2.11 1.1 -4.13 +4.13.1 2.12 2.5.0 6.0.1
[atlas] 03/03: ATLAS-4406: Ignore configs:- A hive table DDL entity is created for ignored hive tables
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit e962143866f69ad96a2207e07a5f0ce1d2e4797e
Author: Disha Talreja
AuthorDate: Thu Aug 26 22:10:08 2021 -0400
ATLAS-4406: Ignore configs:- A hive table DDL entity is created for ignored
hive tables
---
.../org/apache/atlas/notification/preprocessor/HivePreprocessor.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
b/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
index bf6a623..083e343 100644
---
a/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
+++
b/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
@@ -71,6 +71,7 @@ public class HivePreprocessor {
context.addToIgnoredEntities(entity.getAttribute(ATTRIBUTE_SD));
context.addToIgnoredEntities(entity.getAttribute(ATTRIBUTE_COLUMNS));
context.addToIgnoredEntities(entity.getAttribute(ATTRIBUTE_PARTITION_KEYS));
+
context.addToIgnoredEntities(entity.getAttribute(TYPE_HIVE_TABLE_DDL));
} else if (action == PreprocessAction.PRUNE) {
context.addToPrunedEntities(entity);
svn commit: r49546 - in /release/atlas/2.2.0: ./ apache-atlas-2.2.0-sources.tar.gz apache-atlas-2.2.0-sources.tar.gz.asc apache-atlas-2.2.0-sources.tar.gz.md5 apache-atlas-2.2.0-sources.tar.gz.sha512
Author: amestry Date: Tue Aug 17 21:02:25 2021 New Revision: 49546 Log: Atlas 2.2.0 Release Added: release/atlas/2.2.0/ release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz (with props) release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.asc (with props) release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.md5 (with props) release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.sha512 (with props) Added: release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz == Binary file - no diff available. Propchange: release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz -- svn:mime-type = application/octet-stream Added: release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.asc == --- release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.asc (added) +++ release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.asc Tue Aug 17 21:02:25 2021 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCAAdFiEEafAOKx/zDi/OlihRS9P/NuPXXAwFAmERrDEACgkQS9P/NuPX +XAyCfQ/9FCJTyMx6obT1eACnhKE1VPeiANwmKRvxT+dOeClbphS1f0iMesca/3Jv +EekhOKd8IRUWbYLtvFI/zvuG+Bgv/aX/PuAqZP2BztQWtz41AwXczWkeAxzNBOOL +W+QnaZQuhYBE3BA944jD8PV1XxVrmZ+E2JHOwdAOszX4R0budaIG4zp43l44qLOs +vuJj+cpgo4FQ+2oaNPkyEtQy0R7yrMQeJ1B3M5LLF8+3xuo0v5SkUE/kH3ZQ7jEi +QOy4qlcTUajBD5rAzgGZ4aR/c4YFwuALqp+I9tG20mDKTWeo1hwjas9vt+quLkjZ +qpfpElPrAARmcec3aw6zkEZwiHVojXCZ9kUXZ5SqrZb2HviMmLbWlIBBF/waChBj +PKEf0xzpO5Ka4E74VkxLM7u/Wu4vfZDSRZZScrQHF09pPGxOjY5KspFiruILSjg2 +TSxWygiW5xh5IaZ4uGLtz++hCYKra3UezoK8/bfYQwpneIcUsGUX5LhAQ1Da205X +f8q/mhG09duMWcuuRm/sOQFCAhcsFotCeuzfbFb55ZKzCPHhn8QmqUWqTpj/lvy1 +AOO3sFcPTWtQ1ETmEY8ofbWqzIEkht7Mpfv2VExwDqvQ0zYn9ywmVmNM3AAVj8HX +G8qLuDFtl8zMhaVRbxMxsTNymiBkB+4BsGr5WKza4Edr3im8F/U= +=swqi +-END PGP SIGNATURE- Propchange: release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.asc -- svn:eol-style = native Added: release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.md5 == --- release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.md5 (added) +++ release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.md5 Tue Aug 17 21:02:25 2021 @@ -0,0 +1,2 @@ +apache-atlas-2.2.0-sources.tar.gz: 98 29 C3 9E 28 17 12 7F CE 19 87 13 A8 73 11 + 5D Propchange: release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.md5 -- svn:eol-style = native Added: release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.sha512 == --- release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.sha512 (added) +++ release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.sha512 Tue Aug 17 21:02:25 2021 @@ -0,0 +1,4 @@ +apache-atlas-2.2.0-sources.tar.gz: 415B2C5E FF768861 9D911B44 B110B208 331E7912 + 40D3D67B 2F426454 56098030 0ADF39A4 62E2C337 + 413AE87E 26CAD158 B6F94E76 10E57048 F133F65F + 056E866C Propchange: release/atlas/2.2.0/apache-atlas-2.2.0-sources.tar.gz.sha512 -- svn:eol-style = native
svn commit: r49194 - /release/atlas/KEYS
Author: amestry Date: Wed Aug 4 17:32:25 2021 New Revision: 49194 Log: Added signing key for Sidharth Mishra. Modified: release/atlas/KEYS Modified: release/atlas/KEYS == --- release/atlas/KEYS (original) +++ release/atlas/KEYS Wed Aug 4 17:32:25 2021 @@ -437,3 +437,62 @@ ldagWIrb1v5Mi7mFfMPzAP96zwSdjCUVZWwS0Aus AQ== =Rx3t -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2021-08-03 [SC] [expires: 2030-08-03] + 69F00E2B1FF30E2FCE9628514BD3FF36E3D75C0C +uid [ultimate] Sidharth Mishra +sig 34BD3FF36E3D75C0C 2021-08-04 Sidharth Mishra +sub rsa4096 2021-08-03 [E] [expires: 2030-08-03] +sig 4BD3FF36E3D75C0C 2021-08-04 Sidharth Mishra + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBGEJvvYBEAC9RdIStxmuTNYlkRRmk+TVyJww5g1V6UxAobj4FBlqCooOrCZL +Rhh9hDUNAkEuacYJxv6HoFqV3Vt8T6PunmQEltFLgDvv2dUQZVCiopNfhEa1Y+1A +6NmpobOuqRoYim0OjV9EfClRLZ5IwDI1PnAFpEHtgtja9q3bnlObFBLr3uYoZB12 +uwfnG5C1jPFj7ctEGy2OHWqIG6TWIwyB3WuRo9zn4ICVgf0Xo53D5LfD4dwszXn4 +Ve4L2pZR6otqu3KTVtlaUl0O15m1rQEXgxZa8/J++8yrbz7+JNe9lITBMAJSPb40 +gAZd3RD+DSoBS7o0nQRWcI6fKIhOMXebC+VHj3SYsgl18FeAk+wa/vYZ4pzlTKz4 +alFEpbrtFTJ1Thesl1OU+85BZlshX/q3V+S76zHnkwl+/oxhYvKELqj7Wlt2dRRo +Vc/D9AGyCEh3/ERGYPQYrVmQuYJ97G82ygY+PJE2c+EyJhfvfvrzfXo175V+F2cA +yKnnKCMzH8YzqZdbLWAlGD4Y0vrDmdkgzHEeFYJ1Ow5NAcjjTqZ2mhaRMjTqjh/p +sjvt4EpQ1pv1k71TLKyiXsmiW2LCvrNtNpNUwqPuWbLpYGYQTHT2icVMpM6bAPDY +dRM777RHFV3kU0RiFwtGiBqjD6h7PrSkl9Btu6jPEN/87quZWL13DxKy7QARAQAB +tCZTaWRoYXJ0aCBNaXNocmEgPHNpZG1pc2hyYUBhcGFjaGUub3JnPokCVAQTAQgA +PgIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgBYhBGnwDisf8w4vzpYoUUvT/zbj +11wMBQJhCga3BQkQ7XILAAoJEEvT/zbj11wMJEMP/jsHbTShLSpiJcyhnxaS11C6 +jqQJLFyObbhAcFtLCyqgimy6nxSkpJdu7OPXMmBSLXzcTqPZwJNqTS1+bKbhBT8c +NreQ7QhnMUr3SAEoBMKxKJpPlhZs+IPFCmhMJ8ifOG+O/zuCiaUCQyqxmYBcgFh0 +KwV/MnC5Ug6hb0bj+jUsbvOgt9fqA1bcxOSdIxtei86a6pqDYL4ikl3c7IQyQP4N +qVBLxd51EX6+wvdd0hnz6k/sRMNf/e9VnBsJBSsQzU1vPMEDT/9HgqSEp3nEVr6h +IOaAIJ1C6zpi2sDoyfPj4LIzdvUpVS9vSwWYLRaKQCb5XnYeoa1/pjWz+4AVKgQl +ITTNxi7n4nFHzMXpYuQajsCItaK3X1+GR2Y76qj6yZdfq6LsDdrfX5GoeLic19d8 +IZtuz/8kYV8fSkWzsyAgOpXmkRgSo2TrpyRDYPCqAXsMFJ3trzLVUqNv1bzwxmdu +wUtol/C0zK2G71pYsyOcIe1h0Om/1NSzChGswExv6U/fI6r8LqADuS9gpb0wZqC7 +5pCWfEjAFEo9Kg8PcLBHG5NRfhYi1bpfb6FMAV/hn0lcbu341UpU6KXeboHnxnuj +zJtR7616Pswj2H9DB3c1GZCvNMYbJcFfK2WjXe2HZ1Cq0CJiG6KU4WnKOTJhKdED +UZiT1u7kcy8M1+/Pkb0CuQINBGEJvvYBEADahnSq4SRXntifcRWFZe46s3T+HZTF +HtKUKAo0iuuB602bqQnkihXcOCNwvjYJLZ6Djv/tU0ecbS4aSrDfTcUbjrvGH6Pd +2p9JJI5GZeCpGl9aVgJF5Ir0uKdptWaSoFoPQpvyzB0T20/5ZzKpm8QzzxvrsrAw +i/ytTvL+HkSwZj2IGTXkT0WiREAK1JLrb8WyrHppp4ezrRIOy1WC/tZpTHSAImMJ +fLNdTaEmKCdEIsWDY6RxO0SmvFu/UGKshiqLmi7ZfXIFtCIkaNtszB5Zp8inBedq +9KC6m5Yu5hTXriGo6CgzBfgNCQ/H8j5j39b5No1v/BArev8gdtcft5t8qINsLuA+ +e3tbSY0COyjzitInu7dr/ec2dJ9oV/X+QnZYzcBvl3QEtDQbwCaylNrhv5PP1ueL +nBZPMSkvn0sszD0l1aAJf4ECRGnLJxIHGcSvZi2x2aTlx/cjKLUwV7itmm1m7zr6 +CV5UOJJIImfh8PYnVmzBkh8+mjEDwwRMrFznC+XqizDobHlEEKvsNIdgw9EOwXlD +suPUGWZepzE6Ps+/olMVG7O0o/sca/aiDIV9wKc8EP7da7KehU5xbFPVIXDY7bmI +yXQlafqHSo3eAL6OltEIUaIcoO7FDzWGAh/OkUCE4xjoPMtmyaxPOXrDYwvz+32h +udnPoWHLiS0lHwARAQABiQI8BBgBCAAmAhsMFiEEafAOKx/zDi/OlihRS9P/NuPX +XAwFAmEKB9sFCRDtcgsACgkQS9P/NuPXXAxdrA//TgHvFrGTzG+LBGQ/9/7shpVA +a3rpe73495ZXcs1ucRpTzabaHjXmRYGgD6VC11/RGhnW1aqN5YMQZ4Rl7dcl5Olj +USiO3ZxpBuZxpcMpto9Uq9Wbb41a99zgxMdlScRXgaRmA5tnVbJ1FmcRkAeGgT20 +HrfzRoX9XAds5Q3KC0BZftAaiZkUU3csKtu5HsjxMg2B0SoDgDjbigV0FaMt4gfj +tMBmz84+Cbfaz2OYQ5abehILgmPm75Oy7QZjbfcw6wXrTfcidp9xqo2O4tPQ7bd+ +kxVAvcU8Aj5Hak91W56D5QQk2/HSudUATrAjTKZWwS3LGVV1jruo5lszZfojAKCJ +X7bAtfg+MayHpdT9My7nQ9U2968MqJJJ+OgyMwKAJaowm+PxLrMySiPjT/D5xR+Q +P9uQOGOKSrKDAXt29lAX1c+zm80zkkkvcNe5SHe7hk9dbDYpvN37FlLMxh2eaA0k +OPKeKNG0VrYFj/x+osNCkxaZ0mIvdniwTAQa6ZpBaZlTWDE9z0mqrj6rgGbD8oSW +bpwTa6kJR+CV8bf/tdG00kDd8WIDzY77110j7TJc3cPHyM4Jii+mzDokmZvNwlq8 +MxH4NPsOeLEruN87vsPsigCNR7RPJZ82UnaHD/susJka0zB2VDZ5TXNQIHWNwNXf +O72oTUu6olLLCFj9Utw= +=P+MW +-END PGP PUBLIC KEY BLOCK-
svn commit: r49193 - /release/atlas/KEYS
Author: amestry Date: Wed Aug 4 17:05:37 2021 New Revision: 49193 Log: Added keys. Modified: release/atlas/KEYS Modified: release/atlas/KEYS == --- release/atlas/KEYS (original) +++ release/atlas/KEYS Wed Aug 4 17:05:37 2021 @@ -423,3 +423,17 @@ IkQhL7BqVoPKPoA1TzigvOIs9d1M4qKDopmR7ps/ 5NKVyqwikxw/TVkY =5R+g -END PGP PUBLIC KEY BLOCK- +-BEGIN PGP PUBLIC KEY BLOCK- + +mDMEYQoyohYJKwYBBAHaRw8BAQdAK36uaGMCbTiua1E7tcw/q2cT6MKDizZ6dQwZ +xCScGGi0JEFzaHV0b3NoIE1lc3RyeSA8YW1lc3RyeUBhcGFjaGUub3JnPoiaBBMW +CgBCFiEEnioyNlwWOAtrTQz0X6zfV75/vncFAmEKMqICGwMFCQPCZwAFCwkIBwID +IgIBBhUKCQgLAgQWAgMBAh4HAheAAAoJEF+s31e+f753uzcBAKgkcxloIwGjlj0B +rFWCGmekjgRfe3zhVKL6YqXU6NV2AP9zSPy/QYo/LmLfys8ti9tswuhFT8JKEIIv ++04PMXidC7g4BGEKMqISCisGAQQBl1UBBQEBB0B3PwTb+mCPLCTY4my75ydR5rMB +gs6ZnR/5MNnnqr+yJQMBCAeIfgQYFgoAJhYhBJ4qMjZcFjgLa00M9F+s31e+f753 +BQJhCjKiAhsMBQkDwmcAAAoJEF+s31e+f753hSoBAJ51VumEhLBPgR5EzWpbB6UV +ldagWIrb1v5Mi7mFfMPzAP96zwSdjCUVZWwS0AusPV8xJkmxJ5ms+VLTSl2RaA4O +AQ== +=Rx3t +-END PGP PUBLIC KEY BLOCK-
[atlas] 01/02: ATLAS-4372: Upgrade nimbus-jose-jwt to 9.8.1
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git commit 089d2b08c6a9eaba5a06591d3f752d6656dde4bf Author: Disha Talreja AuthorDate: Tue Jul 27 16:48:27 2021 -0400 ATLAS-4372: Upgrade nimbus-jose-jwt to 9.8.1 Signed-off-by: Ashutosh Mestry --- webapp/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/webapp/pom.xml b/webapp/pom.xml index dd5dd82..3785b4e 100755 --- a/webapp/pom.xml +++ b/webapp/pom.xml @@ -439,7 +439,7 @@ com.nimbusds nimbus-jose-jwt -5.4 +9.8.1 compile
[atlas] 02/02: ATLAS-4368: Atlas - Remove elasticsearch dependency or upgrade to 6.8.15
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git commit aee490390a704ddc457529b3d7f7a43421ccbbc4 Author: Disha Talreja AuthorDate: Wed Jul 21 23:07:54 2021 -0400 ATLAS-4368: Atlas - Remove elasticsearch dependency or upgrade to 6.8.15 Signed-off-by: Ashutosh Mestry --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index f23d13c..6609fcb 100644 --- a/pom.xml +++ b/pom.xml @@ -686,7 +686,7 @@ 4.3.0 1.8 3.2.2 -6.8.3 +6.8.15 org.apache.atlas.repository.audit.InMemoryEntityAuditRepository 2.13.2 2.18.1
[atlas] branch branch-2.0 updated (f08a407 -> aee4903)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git. from f08a407 ATLAS-4340: Set Solr wait-searcher property to false by default to make Solr commits async new 089d2b0 ATLAS-4372: Upgrade nimbus-jose-jwt to 9.8.1 new aee4903 ATLAS-4368: Atlas - Remove elasticsearch dependency or upgrade to 6.8.15 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pom.xml| 2 +- webapp/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[atlas] 02/02: ATLAS-4368: Atlas - Remove elasticsearch dependency or upgrade to 6.8.15
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git commit 2576a236913915a928e48edc186d25461dae5774 Author: Disha Talreja AuthorDate: Wed Jul 21 23:07:54 2021 -0400 ATLAS-4368: Atlas - Remove elasticsearch dependency or upgrade to 6.8.15 Signed-off-by: Ashutosh Mestry --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 19db431..532b6c5 100644 --- a/pom.xml +++ b/pom.xml @@ -697,7 +697,7 @@ 4.3.0 1.8 3.2.2 -6.8.3 +6.8.15 org.apache.atlas.repository.audit.InMemoryEntityAuditRepository 2.13.2 2.18.1
[atlas] 01/02: ATLAS-4372: Upgrade nimbus-jose-jwt to 9.8.1
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git commit cd4bb5497c28456c9a816eb7e6e37e1e1029c74d Author: Disha Talreja AuthorDate: Tue Jul 27 16:48:27 2021 -0400 ATLAS-4372: Upgrade nimbus-jose-jwt to 9.8.1 Signed-off-by: Ashutosh Mestry --- webapp/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/webapp/pom.xml b/webapp/pom.xml index ccbcc31..81bbb89 100755 --- a/webapp/pom.xml +++ b/webapp/pom.xml @@ -439,7 +439,7 @@ com.nimbusds nimbus-jose-jwt -5.4 +9.8.1 compile
[atlas] branch master updated (a7ac64d -> 2576a23)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git. from a7ac64d ATLAS-4340: Set Solr wait-searcher property to false by default to make Solr commits async new cd4bb54 ATLAS-4372: Upgrade nimbus-jose-jwt to 9.8.1 new 2576a23 ATLAS-4368: Atlas - Remove elasticsearch dependency or upgrade to 6.8.15 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pom.xml| 2 +- webapp/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[atlas] branch branch-2.0 updated: ATLAS-4324: FS entity created for 'load data inpath' Part 2
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 1ecc609 ATLAS-4324: FS entity created for 'load data inpath' Part 2
1ecc609 is described below
commit 1ecc6099b0a51d4c944741230ee0f99d0f592ca1
Author: Ashutosh Mestry
AuthorDate: Fri Jun 4 13:25:04 2021 -0700
ATLAS-4324: FS entity created for 'load data inpath' Part 2
---
.../java/org/apache/atlas/hive/hook/HiveHook.java | 2 +-
.../atlas/hive/hook/utils/ActiveEntityFilter.java | 24 -
.../atlas/hive/hook/utils/HiveDDLEntityFilter.java | 26 ++---
.../hive/hook/utils/ActiveEntityFilterTest.java| 54 +++
.../src/test/resources/json/hs2-create-db-v2.json | 3 +-
...reate-db-with-no-pathentities-to-retain-v2.json | 30 --
...2-create-db-with-no-pathentities-to-retain.json | 73 --
.../test/resources/json/hs2-create-process-v2.json | 3 -
.../test/resources/json/hs2-create-table-v2.json | 2 -
.../test/resources/json/hs2-load-inpath-v2.json| 108 ++---
...d-inpath-with-no-pathentities-to-retain-v2.json | 4 -
...load-inpath-with-no-pathentities-to-retain.json | 18
.../src/test/resources/json/hs2-load-inpath.json | 108 ++---
.../test/resources/json/hs2-table-rename-v2.json | 6 +-
14 files changed, 238 insertions(+), 223 deletions(-)
diff --git
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
index 9548eda..bf500ca 100644
--- a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
+++ b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
@@ -70,7 +70,7 @@ public class HiveHook extends AtlasHook implements
ExecuteWithHookContext {
public static final String HOOK_HIVE_TABLE_PRUNE_PATTERN
= CONF_PREFIX + "hive_table.prune.pattern";
public static final String HOOK_HIVE_TABLE_CACHE_SIZE
= CONF_PREFIX + "hive_table.cache.size";
public static final String HOOK_HIVE_IGNORE_DDL_OPERATIONS
= CONF_PREFIX + "hs2.ignore.ddl.operations";
-public static final String HOOK_HIVE_FILTER_ENTITY_TYPES_TO_RETAIN
= CONF_PREFIX + "hs2.filter.entity.types.to.retain";
+public static final String
HOOK_HIVE_FILTER_ENTITY_ADDITIONAL_TYPES_TO_RETAIN = CONF_PREFIX +
"hs2.filter.entity.additional.types.to.retain";
public static final String DEFAULT_HOST_NAME = "localhost";
private static final Map OPERATION_MAP = new
HashMap<>();
diff --git
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
index 737c637..0b0d4d6 100644
---
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
+++
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
@@ -18,11 +18,13 @@
package org.apache.atlas.hive.hook.utils;
import com.google.common.annotations.VisibleForTesting;
+import org.apache.atlas.hive.hook.HiveHook;
import org.apache.atlas.model.notification.HookNotification;
import org.apache.commons.configuration.Configuration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.Arrays;
import java.util.List;
import static
org.apache.atlas.hive.hook.HiveHook.HOOK_HIVE_IGNORE_DDL_OPERATIONS;
@@ -34,16 +36,32 @@ public class ActiveEntityFilter {
public static void init(Configuration configuration) {
boolean skipDdlOperations =
configuration.getBoolean(HOOK_HIVE_IGNORE_DDL_OPERATIONS, false);
-init(skipDdlOperations);
+List additionalTypesToRetain =
getConfiguredTypesToRetainForDDLEntityFilter(configuration);
+
+init(skipDdlOperations, additionalTypesToRetain);
LOG.info("atlas.hook.hive.ignore.ddl.operations={} - {}",
skipDdlOperations, entityFilter.getClass().getSimpleName());
}
@VisibleForTesting
-static void init(boolean lineageOnlyFilter) {
-entityFilter = lineageOnlyFilter ? new HiveDDLEntityFilter() : new
PassthroughFilter();
+static void init(boolean lineageOnlyFilter, List
additionalTypesToRetain) {
+entityFilter = lineageOnlyFilter ? new
HiveDDLEntityFilter(additionalTypesToRetain) : new PassthroughFilter();
}
public static List apply(List
incoming) {
return entityFilter.apply(incoming);
}
+
+private static List
getConfiguredTypesToRetainForDDLEntityFilter(Configuration configuration) {
+try {
+if
(configuration.containsKey(HiveHook.HOO
[atlas] branch master updated: ATLAS-4324: FS entity created for 'load data inpath' Part 2
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 0ac4b12 ATLAS-4324: FS entity created for 'load data inpath' Part 2
0ac4b12 is described below
commit 0ac4b12be7c38ba34951da6bdbe82d23968cce13
Author: Ashutosh Mestry
AuthorDate: Fri Jun 4 13:25:04 2021 -0700
ATLAS-4324: FS entity created for 'load data inpath' Part 2
---
.../java/org/apache/atlas/hive/hook/HiveHook.java | 2 +-
.../atlas/hive/hook/utils/ActiveEntityFilter.java | 24 -
.../atlas/hive/hook/utils/HiveDDLEntityFilter.java | 26 ++---
.../hive/hook/utils/ActiveEntityFilterTest.java| 54 +++
.../src/test/resources/json/hs2-create-db-v2.json | 3 +-
...reate-db-with-no-pathentities-to-retain-v2.json | 30 --
...2-create-db-with-no-pathentities-to-retain.json | 73 --
.../test/resources/json/hs2-create-process-v2.json | 3 -
.../test/resources/json/hs2-create-table-v2.json | 2 -
.../test/resources/json/hs2-load-inpath-v2.json| 108 ++---
...d-inpath-with-no-pathentities-to-retain-v2.json | 4 -
...load-inpath-with-no-pathentities-to-retain.json | 18
.../src/test/resources/json/hs2-load-inpath.json | 108 ++---
.../test/resources/json/hs2-table-rename-v2.json | 6 +-
14 files changed, 238 insertions(+), 223 deletions(-)
diff --git
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
index 94ef225..37084d5 100644
--- a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
+++ b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
@@ -70,7 +70,7 @@ public class HiveHook extends AtlasHook implements
ExecuteWithHookContext {
public static final String HOOK_HIVE_TABLE_PRUNE_PATTERN
= CONF_PREFIX + "hive_table.prune.pattern";
public static final String HOOK_HIVE_TABLE_CACHE_SIZE
= CONF_PREFIX + "hive_table.cache.size";
public static final String HOOK_HIVE_IGNORE_DDL_OPERATIONS
= CONF_PREFIX + "hs2.ignore.ddl.operations";
-public static final String HOOK_HIVE_FILTER_ENTITY_TYPES_TO_RETAIN
= CONF_PREFIX + "hs2.filter.entity.types.to.retain";
+public static final String
HOOK_HIVE_FILTER_ENTITY_ADDITIONAL_TYPES_TO_RETAIN = CONF_PREFIX +
"hs2.filter.entity.additional.types.to.retain";
public static final String DEFAULT_HOST_NAME = "localhost";
private static final Map OPERATION_MAP = new
HashMap<>();
diff --git
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
index 737c637..0b0d4d6 100644
---
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
+++
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
@@ -18,11 +18,13 @@
package org.apache.atlas.hive.hook.utils;
import com.google.common.annotations.VisibleForTesting;
+import org.apache.atlas.hive.hook.HiveHook;
import org.apache.atlas.model.notification.HookNotification;
import org.apache.commons.configuration.Configuration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.Arrays;
import java.util.List;
import static
org.apache.atlas.hive.hook.HiveHook.HOOK_HIVE_IGNORE_DDL_OPERATIONS;
@@ -34,16 +36,32 @@ public class ActiveEntityFilter {
public static void init(Configuration configuration) {
boolean skipDdlOperations =
configuration.getBoolean(HOOK_HIVE_IGNORE_DDL_OPERATIONS, false);
-init(skipDdlOperations);
+List additionalTypesToRetain =
getConfiguredTypesToRetainForDDLEntityFilter(configuration);
+
+init(skipDdlOperations, additionalTypesToRetain);
LOG.info("atlas.hook.hive.ignore.ddl.operations={} - {}",
skipDdlOperations, entityFilter.getClass().getSimpleName());
}
@VisibleForTesting
-static void init(boolean lineageOnlyFilter) {
-entityFilter = lineageOnlyFilter ? new HiveDDLEntityFilter() : new
PassthroughFilter();
+static void init(boolean lineageOnlyFilter, List
additionalTypesToRetain) {
+entityFilter = lineageOnlyFilter ? new
HiveDDLEntityFilter(additionalTypesToRetain) : new PassthroughFilter();
}
public static List apply(List
incoming) {
return entityFilter.apply(incoming);
}
+
+private static List
getConfiguredTypesToRetainForDDLEntityFilter(Configuration configuration) {
+try {
+if
(configuration.containsKey(HiveHook.HOO
[atlas] branch branch-2.0 updated: ATLAS-4333: [MATERIALIZED VIEW]Column Lineage and hive_process missing in case of CREATE MATERIALIZED VIEW query at Hive
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 62c7024 ATLAS-4333: [MATERIALIZED VIEW]Column Lineage and
hive_process missing in case of CREATE MATERIALIZED VIEW query at Hive
62c7024 is described below
commit 62c702422f7523b662743a5e2de15ecef6a27c3c
Author: Radhika Kundam
AuthorDate: Thu Jun 10 09:49:05 2021 -0700
ATLAS-4333: [MATERIALIZED VIEW]Column Lineage and hive_process missing in
case of CREATE MATERIALIZED VIEW query at Hive
---
.../main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
index 659417d..e5295f4 100644
---
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
+++
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
@@ -292,6 +292,7 @@ public class CreateHiveProcess extends BaseHiveEvent {
return entity != null && !context.isMetastoreHook()
&&
(context.getHiveOperation().equals(HiveOperation.CREATETABLE_AS_SELECT)
|| context.getHiveOperation().equals(HiveOperation.CREATEVIEW)
- || context.getHiveOperation().equals(HiveOperation.ALTERVIEW_AS));
+ || context.getHiveOperation().equals(HiveOperation.ALTERVIEW_AS)
+ ||
context.getHiveOperation().equals(HiveOperation.CREATE_MATERIALIZED_VIEW));
}
}
[atlas] branch master updated: ATLAS-4333: [MATERIALIZED VIEW]Column Lineage and hive_process missing in case of CREATE MATERIALIZED VIEW query at Hive
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 01f5eea ATLAS-4333: [MATERIALIZED VIEW]Column Lineage and
hive_process missing in case of CREATE MATERIALIZED VIEW query at Hive
01f5eea is described below
commit 01f5eea5c60b710e16370b7e09a60a5353f8d17d
Author: Radhika Kundam
AuthorDate: Thu Jun 10 09:49:05 2021 -0700
ATLAS-4333: [MATERIALIZED VIEW]Column Lineage and hive_process missing in
case of CREATE MATERIALIZED VIEW query at Hive
---
.../main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
index 659417d..e5295f4 100644
---
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
+++
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
@@ -292,6 +292,7 @@ public class CreateHiveProcess extends BaseHiveEvent {
return entity != null && !context.isMetastoreHook()
&&
(context.getHiveOperation().equals(HiveOperation.CREATETABLE_AS_SELECT)
|| context.getHiveOperation().equals(HiveOperation.CREATEVIEW)
- || context.getHiveOperation().equals(HiveOperation.ALTERVIEW_AS));
+ || context.getHiveOperation().equals(HiveOperation.ALTERVIEW_AS)
+ ||
context.getHiveOperation().equals(HiveOperation.CREATE_MATERIALIZED_VIEW));
}
}
[atlas] branch branch-2.0 updated: ATLAS-4313 : ATLAS-4311: Business Metadata Bulk Import - Issue with multiple values for Date Type & wrong line number
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new ded2721 ATLAS-4313 : ATLAS-4311: Business Metadata Bulk Import -
Issue with multiple values for Date Type & wrong line number
ded2721 is described below
commit ded2721ce3298f15f9a304207cf38629ffd362ef
Author: Radhika Kundam
AuthorDate: Wed Jun 2 13:25:33 2021 -0700
ATLAS-4313 : ATLAS-4311: Business Metadata Bulk Import - Issue with
multiple values for Date Type & wrong line number
---
.../repository/store/graph/v2/AtlasEntityStoreV2.java | 17 +
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
index 0f12bd3..d0e95e4 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
@@ -1564,7 +1564,8 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
List failedMsgList = new ArrayList<>();
for (int lineIndex = 0; lineIndex < fileData.size(); lineIndex++) {
-String[] record = fileData.get(lineIndex);
+String[] record = fileData.get(lineIndex);
+int lineIndexToLog = lineIndex + 2;
boolean missingFields = record.length <
FileUtils.UNIQUE_ATTR_NAME_COLUMN_INDEX ||
StringUtils.isBlank(record[FileUtils.TYPENAME_COLUMN_INDEX]) ||
@@ -1573,7 +1574,7 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
StringUtils.isBlank(record[FileUtils.BM_ATTR_VALUE_COLUMN_INDEX]);
if (missingFields){
-failedMsgList.add("Line #" + (lineIndex + 1) + ": missing
fields. " + Arrays.toString(record));
+failedMsgList.add("Line #" + lineIndexToLog + ": missing
fields. " + Arrays.toString(record));
continue;
}
@@ -1582,7 +1583,7 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
AtlasEntityType entityType =
typeRegistry.getEntityTypeByName(typeName);
if (entityType == null) {
-failedMsgList.add("Line #" + (lineIndex + 1) + ": invalid
entity-type '" + typeName + "'");
+failedMsgList.add("Line #" + lineIndexToLog + ": invalid
entity-type '" + typeName + "'");
continue;
}
@@ -1599,13 +1600,13 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
AtlasAttribute uniqueAttribute =
entityType.getAttribute(uniqueAttrName);
if (uniqueAttribute == null) {
-failedMsgList.add("Line #" + (lineIndex + 1) + ": attribute '"
+ uniqueAttrName + "' not found in entity-type '" + typeName + "'");
+failedMsgList.add("Line #" + lineIndexToLog + ": attribute '"
+ uniqueAttrName + "' not found in entity-type '" + typeName + "'");
continue;
}
if (!uniqueAttribute.getAttributeDef().getIsUnique()) {
-failedMsgList.add("Line #" + (lineIndex + 1) + ": attribute '"
+ uniqueAttrName + "' is not an unique attribute in entity-type '" + typeName +
"'");
+failedMsgList.add("Line #" + lineIndexToLog + ": attribute '"
+ uniqueAttrName + "' is not an unique attribute in entity-type '" + typeName +
"'");
continue;
}
@@ -1617,7 +1618,7 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
vertex =
AtlasGraphUtilsV2.findByTypeAndUniquePropertyName(graph, typeName,
uniqueAttribute.getVertexUniquePropertyName(), uniqueAttrValue);
if (vertex == null) {
-failedMsgList.add("Line #" + (lineIndex + 1) + ": no " +
typeName + " entity found with " + uniqueAttrName + "=" + uniqueAttrValue);
+failedMsgList.add("Line #" + lineIndexToLog + ": no " +
typeName + " entity found with " + uniqueAttrName + "=" + uniqueAttrValue);
continue;
}
@@ -1628,7 +1629,7 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
AtlasBusinessAttribute businessAttribute =
entityType.getBusinesAAttribute(b
[atlas] branch master updated: ATLAS-4313 : ATLAS-4311: Business Metadata Bulk Import - Issue with multiple values for Date Type & wrong line number
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new edf9936 ATLAS-4313 : ATLAS-4311: Business Metadata Bulk Import -
Issue with multiple values for Date Type & wrong line number
edf9936 is described below
commit edf9936ce9692011a69be2cd58817e70942aa574
Author: Radhika Kundam
AuthorDate: Wed Jun 2 13:25:33 2021 -0700
ATLAS-4313 : ATLAS-4311: Business Metadata Bulk Import - Issue with
multiple values for Date Type & wrong line number
---
.../repository/store/graph/v2/AtlasEntityStoreV2.java | 17 +
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
index 0f12bd3..d0e95e4 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
@@ -1564,7 +1564,8 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
List failedMsgList = new ArrayList<>();
for (int lineIndex = 0; lineIndex < fileData.size(); lineIndex++) {
-String[] record = fileData.get(lineIndex);
+String[] record = fileData.get(lineIndex);
+int lineIndexToLog = lineIndex + 2;
boolean missingFields = record.length <
FileUtils.UNIQUE_ATTR_NAME_COLUMN_INDEX ||
StringUtils.isBlank(record[FileUtils.TYPENAME_COLUMN_INDEX]) ||
@@ -1573,7 +1574,7 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
StringUtils.isBlank(record[FileUtils.BM_ATTR_VALUE_COLUMN_INDEX]);
if (missingFields){
-failedMsgList.add("Line #" + (lineIndex + 1) + ": missing
fields. " + Arrays.toString(record));
+failedMsgList.add("Line #" + lineIndexToLog + ": missing
fields. " + Arrays.toString(record));
continue;
}
@@ -1582,7 +1583,7 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
AtlasEntityType entityType =
typeRegistry.getEntityTypeByName(typeName);
if (entityType == null) {
-failedMsgList.add("Line #" + (lineIndex + 1) + ": invalid
entity-type '" + typeName + "'");
+failedMsgList.add("Line #" + lineIndexToLog + ": invalid
entity-type '" + typeName + "'");
continue;
}
@@ -1599,13 +1600,13 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
AtlasAttribute uniqueAttribute =
entityType.getAttribute(uniqueAttrName);
if (uniqueAttribute == null) {
-failedMsgList.add("Line #" + (lineIndex + 1) + ": attribute '"
+ uniqueAttrName + "' not found in entity-type '" + typeName + "'");
+failedMsgList.add("Line #" + lineIndexToLog + ": attribute '"
+ uniqueAttrName + "' not found in entity-type '" + typeName + "'");
continue;
}
if (!uniqueAttribute.getAttributeDef().getIsUnique()) {
-failedMsgList.add("Line #" + (lineIndex + 1) + ": attribute '"
+ uniqueAttrName + "' is not an unique attribute in entity-type '" + typeName +
"'");
+failedMsgList.add("Line #" + lineIndexToLog + ": attribute '"
+ uniqueAttrName + "' is not an unique attribute in entity-type '" + typeName +
"'");
continue;
}
@@ -1617,7 +1618,7 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
vertex =
AtlasGraphUtilsV2.findByTypeAndUniquePropertyName(graph, typeName,
uniqueAttribute.getVertexUniquePropertyName(), uniqueAttrValue);
if (vertex == null) {
-failedMsgList.add("Line #" + (lineIndex + 1) + ": no " +
typeName + " entity found with " + uniqueAttrName + "=" + uniqueAttrValue);
+failedMsgList.add("Line #" + lineIndexToLog + ": no " +
typeName + " entity found with " + uniqueAttrName + "=" + uniqueAttrValue);
continue;
}
@@ -1628,7 +1629,7 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
AtlasBusinessAttribute businessAttribute =
entityType.getBusinesAAttribute(b
[atlas] branch master updated: ATLAS-4306: Support for User-specific Spool Directory
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 36678ab ATLAS-4306: Support for User-specific Spool Directory
36678ab is described below
commit 36678ab1f331eb717578c3b2ed6677544ac3aa2a
Author: Ashutosh Mestry
AuthorDate: Mon May 31 21:34:44 2021 -0700
ATLAS-4306: Support for User-specific Spool Directory
---
.../atlas/notification/spool/AtlasFileSpool.java | 4 ++-
.../atlas/notification/spool/IndexManagement.java | 6 ++---
.../notification/spool/SpoolConfiguration.java | 22 +---
.../atlas/notification/spool/SpoolUtils.java | 30 +-
.../apache/atlas/notification/spool/BaseTest.java | 2 +-
5 files changed, 54 insertions(+), 10 deletions(-)
diff --git
a/notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
b/notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
index ea31284..0c92c30 100644
---
a/notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
+++
b/notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
@@ -41,6 +41,7 @@ public class AtlasFileSpool implements NotificationInterface {
private final Publisherpublisher;
private Thread publisherThread;
private Boolean initDone = null;
+private String currentUser;
public AtlasFileSpool(Configuration configuration, AbstractNotification
notificationHandler) {
this.notificationHandler = notificationHandler;
@@ -56,7 +57,7 @@ public class AtlasFileSpool implements NotificationInterface {
if (!isInitDone()) {
try {
-config.setSource(source);
+config.setSource(source, this.currentUser);
LOG.info("{}: Initialization: Starting...",
this.config.getSourceName());
@@ -86,6 +87,7 @@ public class AtlasFileSpool implements NotificationInterface {
@Override
public void setCurrentUser(String user) {
this.notificationHandler.setCurrentUser(user);
+this.currentUser = user;
}
@Override
diff --git
a/notification/src/main/java/org/apache/atlas/notification/spool/IndexManagement.java
b/notification/src/main/java/org/apache/atlas/notification/spool/IndexManagement.java
index f018983..adbb8d1 100644
---
a/notification/src/main/java/org/apache/atlas/notification/spool/IndexManagement.java
+++
b/notification/src/main/java/org/apache/atlas/notification/spool/IndexManagement.java
@@ -54,14 +54,14 @@ public class IndexManagement {
public void init() throws IOException, AtlasException {
String sourceName = config.getSourceName();
-File spoolDir = SpoolUtils.getCreateDirectory(config.getSpoolDir());
-
+File spoolDir =
SpoolUtils.getCreateDirectoryWithPermissionCheck(config.getSpoolDir(),
config.getUser());
if (spoolDir == null) {
throw new AtlasException(String.format("%s: %s not found or
inaccessible!", sourceName, spoolDir.getAbsolutePath()));
}
-File archiveDir =
SpoolUtils.getCreateDirectory(config.getArchiveDir());
+config.setSpoolDir(spoolDir.getAbsolutePath());
+File archiveDir =
SpoolUtils.getCreateDirectory(config.getArchiveDir());
if (archiveDir == null) {
throw new AtlasException(String.format("%s: %s not found or
inaccessible!", sourceName, archiveDir.getAbsolutePath()));
}
diff --git
a/notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
b/notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
index 76f05ef..36ea7be 100644
---
a/notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
+++
b/notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
@@ -40,19 +40,23 @@ public class SpoolConfiguration {
public static final String PROP_FILE_SPOOL_PAUSE_BEFORE_SEND_SEC
= PROPERTY_PREFIX_SPOOL + "pause.before.send.sec";
private static final String PROP_HIVE_METASTORE_NAME
= PROPERTY_PREFIX_SPOOL + "hivemetastore.name";
+private final Configuration config;
+
private final String messageHandlerName;
private final intmaxArchivedFilesCount;
private final intmessageBatchSize;
private final intretryDestinationMS;
private final intfileRollOverSec;
private final intfileSpoolMaxFilesCount;
-private final String spoolDirPath;
-private final String archiveDir;
+private String spoolDirPath;
+private String archiveDir;
private final intpauseBeforeSe
[atlas] branch branch-2.0 updated: ATLAS-4310: Handled NPE for DELETE classification.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 4e0228a ATLAS-4310: Handled NPE for DELETE classification.
4e0228a is described below
commit 4e0228ac7030e4f76c98b86c0625f984731c00fe
Author: Ashutosh Mestry
AuthorDate: Wed May 26 13:58:03 2021 -0700
ATLAS-4310: Handled NPE for DELETE classification.
---
.../org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
index 5baff33..b51cbfc 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
@@ -2508,13 +2508,13 @@ public class EntityGraphMapper {
}
AtlasVertex classificationVertex =
graph.getVertex(classificationVertexId);
-AtlasClassification classification =
entityRetriever.toAtlasClassification(classificationVertex);
if (classificationVertex == null) {
LOG.warn("deleteClassificationPropagation(classificationVertexId={}):
classification vertex not found", classificationVertexId);
return null;
}
+AtlasClassification classification =
entityRetriever.toAtlasClassification(classificationVertex);
List entityVertices =
deleteDelegate.getHandler().removeTagPropagation(classificationVertex);
deleteDelegate.getHandler().deleteClassificationVertex(classificationVertex,
true);
if (CollectionUtils.isEmpty(entityVertices)) {
[atlas] branch master updated: ATLAS-4310: Handled NPE for DELETE classification.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new fd26e12 ATLAS-4310: Handled NPE for DELETE classification.
fd26e12 is described below
commit fd26e12f689161fd85676b4a3d81cf1fcbd4e10a
Author: Ashutosh Mestry
AuthorDate: Wed May 26 13:58:03 2021 -0700
ATLAS-4310: Handled NPE for DELETE classification.
---
.../org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
index 5baff33..b51cbfc 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
@@ -2508,13 +2508,13 @@ public class EntityGraphMapper {
}
AtlasVertex classificationVertex =
graph.getVertex(classificationVertexId);
-AtlasClassification classification =
entityRetriever.toAtlasClassification(classificationVertex);
if (classificationVertex == null) {
LOG.warn("deleteClassificationPropagation(classificationVertexId={}):
classification vertex not found", classificationVertexId);
return null;
}
+AtlasClassification classification =
entityRetriever.toAtlasClassification(classificationVertex);
List entityVertices =
deleteDelegate.getHandler().removeTagPropagation(classificationVertex);
deleteDelegate.getHandler().deleteClassificationVertex(classificationVertex,
true);
if (CollectionUtils.isEmpty(entityVertices)) {
[atlas] branch branch-2.0 updated: ATLAS-4272: Changed the cache to thread local and clearing after import
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new c470a5c ATLAS-4272: Changed the cache to thread local and clearing
after import
c470a5c is described below
commit c470a5cb454c9da46570479f960b3dd79a00b944
Author: sidmishra
AuthorDate: Mon May 3 14:59:10 2021 -0700
ATLAS-4272: Changed the cache to thread local and clearing after import
---
.../java/org/apache/atlas/glossary/GlossaryService.java| 14 +-
.../java/org/apache/atlas/glossary/GlossaryTermUtils.java | 13 +
2 files changed, 18 insertions(+), 9 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
index be757ad..73217de 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
@@ -1125,13 +1125,17 @@ public class GlossaryService {
throw new AtlasBaseException(AtlasErrorCode.INVALID_FILE_TYPE,
fileName);
}
-List fileData = FileUtils.readFileData(fileName,
inputStream);
+try {
+List fileData = FileUtils.readFileData(fileName,
inputStream);
-List glossaryTermsWithoutRelations =
glossaryTermUtils.getGlossaryTermDataWithoutRelations(fileData, ret);
-createGlossaryTerms(glossaryTermsWithoutRelations, ret);
+List glossaryTermsWithoutRelations =
glossaryTermUtils.getGlossaryTermDataWithoutRelations(fileData, ret);
+createGlossaryTerms(glossaryTermsWithoutRelations, ret);
-List glossaryTermsWithRelations =
glossaryTermUtils.getGlossaryTermDataWithRelations(fileData, ret);
-updateGlossaryTermsRelation(glossaryTermsWithRelations, ret);
+List glossaryTermsWithRelations =
glossaryTermUtils.getGlossaryTermDataWithRelations(fileData, ret);
+updateGlossaryTermsRelation(glossaryTermsWithRelations, ret);
+} finally {
+glossaryTermUtils.clearImportCache();
+}
return ret;
}
diff --git
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
index dc39fd2..553d3d0 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
@@ -49,6 +49,7 @@ import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
+import java.util.LinkedHashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
@@ -65,7 +66,7 @@ public class GlossaryTermUtils extends GlossaryUtils {
private static final int INDEX_FOR_GLOSSARY_AT_RECORD = 0;
private static final int INDEX_FOR_TERM_AT_RECORD = 1;
-Map glossaryNameGuidCacheForImport = new HashMap<>();
+private static final ThreadLocal>
glossaryNameGuidCacheForImport = ThreadLocal.withInitial(() -> new
LinkedHashMap<>());
protected GlossaryTermUtils(AtlasRelationshipStore relationshipStore,
AtlasTypeRegistry typeRegistry, DataAccess dataAccess) {
super(relationshipStore, typeRegistry, dataAccess);
@@ -148,6 +149,10 @@ public class GlossaryTermUtils extends GlossaryUtils {
}
}
+public void clearImportCache() {
+glossaryNameGuidCacheForImport.get().clear();
+}
+
private boolean isRelationshipGuidSame(AtlasRelatedObjectId storeObject,
AtlasRelatedObjectId relatedObjectId) {
return StringUtils.equals(relatedObjectId.getRelationshipGuid(),
storeObject.getRelationshipGuid());
}
@@ -555,7 +560,7 @@ public class GlossaryTermUtils extends GlossaryUtils {
} else {
glossaryName = record[INDEX_FOR_GLOSSARY_AT_RECORD];
-String glossaryGuid =
glossaryNameGuidCacheForImport.get(glossaryName);
+String glossaryGuid =
glossaryNameGuidCacheForImport.get().get(glossaryName);
if (StringUtils.isEmpty(glossaryGuid)) {
glossaryGuid = getGlossaryGUIDFromGraphDB(glossaryName);
@@ -564,7 +569,7 @@ public class GlossaryTermUtils extends GlossaryUtils {
glossaryGuid = createGlossary(glossaryName,
failedTermMsgs);
}
-glossaryNameGuidCacheForImport.put(glossaryName,
glossaryGuid);
+glossaryNameGuidCacheForImport.get().put(glossaryName,
glossaryGuid);
}
if (StringUtils.isNotEmpty(glossaryGuid)) {
@@ -599,7 +604,7 @@ public class GlossaryTermUtils extends GlossaryUtils {
if (ArrayUtils
[atlas] branch master updated: ATLAS-4272: Changed the cache to thread local and clearing after import
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 77914a3 ATLAS-4272: Changed the cache to thread local and clearing
after import
77914a3 is described below
commit 77914a3a0dafc0975ed5654ea5394d4215e2d426
Author: sidmishra
AuthorDate: Mon May 3 14:59:10 2021 -0700
ATLAS-4272: Changed the cache to thread local and clearing after import
---
.../java/org/apache/atlas/glossary/GlossaryService.java| 14 +-
.../java/org/apache/atlas/glossary/GlossaryTermUtils.java | 13 +
2 files changed, 18 insertions(+), 9 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
index be757ad..73217de 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
@@ -1125,13 +1125,17 @@ public class GlossaryService {
throw new AtlasBaseException(AtlasErrorCode.INVALID_FILE_TYPE,
fileName);
}
-List fileData = FileUtils.readFileData(fileName,
inputStream);
+try {
+List fileData = FileUtils.readFileData(fileName,
inputStream);
-List glossaryTermsWithoutRelations =
glossaryTermUtils.getGlossaryTermDataWithoutRelations(fileData, ret);
-createGlossaryTerms(glossaryTermsWithoutRelations, ret);
+List glossaryTermsWithoutRelations =
glossaryTermUtils.getGlossaryTermDataWithoutRelations(fileData, ret);
+createGlossaryTerms(glossaryTermsWithoutRelations, ret);
-List glossaryTermsWithRelations =
glossaryTermUtils.getGlossaryTermDataWithRelations(fileData, ret);
-updateGlossaryTermsRelation(glossaryTermsWithRelations, ret);
+List glossaryTermsWithRelations =
glossaryTermUtils.getGlossaryTermDataWithRelations(fileData, ret);
+updateGlossaryTermsRelation(glossaryTermsWithRelations, ret);
+} finally {
+glossaryTermUtils.clearImportCache();
+}
return ret;
}
diff --git
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
index dc39fd2..553d3d0 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
@@ -49,6 +49,7 @@ import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
+import java.util.LinkedHashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
@@ -65,7 +66,7 @@ public class GlossaryTermUtils extends GlossaryUtils {
private static final int INDEX_FOR_GLOSSARY_AT_RECORD = 0;
private static final int INDEX_FOR_TERM_AT_RECORD = 1;
-Map glossaryNameGuidCacheForImport = new HashMap<>();
+private static final ThreadLocal>
glossaryNameGuidCacheForImport = ThreadLocal.withInitial(() -> new
LinkedHashMap<>());
protected GlossaryTermUtils(AtlasRelationshipStore relationshipStore,
AtlasTypeRegistry typeRegistry, DataAccess dataAccess) {
super(relationshipStore, typeRegistry, dataAccess);
@@ -148,6 +149,10 @@ public class GlossaryTermUtils extends GlossaryUtils {
}
}
+public void clearImportCache() {
+glossaryNameGuidCacheForImport.get().clear();
+}
+
private boolean isRelationshipGuidSame(AtlasRelatedObjectId storeObject,
AtlasRelatedObjectId relatedObjectId) {
return StringUtils.equals(relatedObjectId.getRelationshipGuid(),
storeObject.getRelationshipGuid());
}
@@ -555,7 +560,7 @@ public class GlossaryTermUtils extends GlossaryUtils {
} else {
glossaryName = record[INDEX_FOR_GLOSSARY_AT_RECORD];
-String glossaryGuid =
glossaryNameGuidCacheForImport.get(glossaryName);
+String glossaryGuid =
glossaryNameGuidCacheForImport.get().get(glossaryName);
if (StringUtils.isEmpty(glossaryGuid)) {
glossaryGuid = getGlossaryGUIDFromGraphDB(glossaryName);
@@ -564,7 +569,7 @@ public class GlossaryTermUtils extends GlossaryUtils {
glossaryGuid = createGlossary(glossaryName,
failedTermMsgs);
}
-glossaryNameGuidCacheForImport.put(glossaryName,
glossaryGuid);
+glossaryNameGuidCacheForImport.get().put(glossaryName,
glossaryGuid);
}
if (StringUtils.isNotEmpty(glossaryGuid)) {
@@ -599,7 +604,7 @@ public class GlossaryTermUtils extends GlossaryUtils {
if (ArrayUtils.isNotEmpty(record) &&am
[atlas] branch branch-2.0 updated: ATLAS-4204: Configure HS2 hook to send lineage-only messages. Part-2
This is an automated email from the ASF dual-hosted git repository. amestry pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new 6a7e2e4 ATLAS-4204: Configure HS2 hook to send lineage-only messages. Part-2 6a7e2e4 is described below commit 6a7e2e41188ca8545c061f34e1b16fbe84c46a8d Author: Ashutosh Mestry AuthorDate: Tue Apr 13 09:25:19 2021 -0700 ATLAS-4204: Configure HS2 hook to send lineage-only messages. Part-2 --- addons/hive-bridge/src/test/resources/atlas-application.properties | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/addons/hive-bridge/src/test/resources/atlas-application.properties b/addons/hive-bridge/src/test/resources/atlas-application.properties index 2848ceb..5d24a30 100644 --- a/addons/hive-bridge/src/test/resources/atlas-application.properties +++ b/addons/hive-bridge/src/test/resources/atlas-application.properties @@ -122,4 +122,4 @@ atlas.authentication.method.file=true atlas.authentication.method.ldap.type=none atlas.authentication.method.kerberos=false # atlas.authentication.method.file.filename=users-credentials.properties -atlas.hook.hive.ignore.ddl.operations=false \ No newline at end of file +atlas.hook.hive.hs2.ignore.ddl.operations=false \ No newline at end of file
[atlas] branch branch-2.0 updated: ATLAS-4204: Configure HS2 hook to send lineage-only messages.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 72475c3 ATLAS-4204: Configure HS2 hook to send lineage-only messages.
72475c3 is described below
commit 72475c3a929b8d913ac1758bcfd4db920c27eb82
Author: Ashutosh Mestry
AuthorDate: Mon Apr 12 09:19:42 2021 -0700
ATLAS-4204: Configure HS2 hook to send lineage-only messages.
---
.../atlas/hive/hook/AtlasHiveHookContext.java | 4 +
.../java/org/apache/atlas/hive/hook/HiveHook.java | 8 +-
.../atlas/hive/hook/events/BaseHiveEvent.java | 2 +
.../atlas/hive/hook/utils/ActiveEntityFilter.java | 49
.../apache/atlas/hive/hook/utils/EntityFilter.java | 26 ++
.../atlas/hive/hook/utils/HiveDDLEntityFilter.java | 258 +++
.../atlas/hive/hook/utils/PassthroughFilter.java | 29 +++
.../hive/hook/utils/ActiveEntityFilterTest.java| 204 +++
.../test/resources/atlas-application.properties| 1 +
.../src/test/resources/json/hs2-alter-view-v2.json | 35 +++
.../src/test/resources/json/hs2-alter-view.json| 268 +++
.../src/test/resources/json/hs2-create-db-v2.json | 30 +++
.../src/test/resources/json/hs2-create-db.json | 73 ++
.../test/resources/json/hs2-create-process-v2.json | 144 +++
.../test/resources/json/hs2-create-process.json| 283 +
.../test/resources/json/hs2-create-table-v2.json | 102
.../src/test/resources/json/hs2-create-table.json | 255 +++
.../src/test/resources/json/hs2-drop-db-v2.json| 4 +
.../src/test/resources/json/hs2-drop-db.json | 10 +
.../src/test/resources/json/hs2-drop-table-v2.json | 1 +
.../src/test/resources/json/hs2-drop-table.json| 50
.../test/resources/json/hs2-table-rename-v2.json | 35 +++
.../src/test/resources/json/hs2-table-rename.json | 276
.../main/java/org/apache/atlas/type/AtlasType.java | 3 +
.../java/org/apache/atlas/utils/AtlasJson.java | 14 +
25 files changed, 2162 insertions(+), 2 deletions(-)
diff --git
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
index 1286471..ab8e183 100644
---
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
+++
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
@@ -246,6 +246,10 @@ public class AtlasHiveHookContext {
}
private void init() {
+if (hiveOperation == null) {
+return;
+}
+
String operation = hiveOperation.getOperationName();
if (knownObjects == null || !isCreateAlterOperation(operation)) {
diff --git
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
index 3b0ee6e..5e6785a 100644
--- a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
+++ b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
@@ -19,6 +19,7 @@
package org.apache.atlas.hive.hook;
import org.apache.atlas.hive.hook.events.*;
+import org.apache.atlas.hive.hook.utils.ActiveEntityFilter;
import org.apache.atlas.hook.AtlasHook;
import org.apache.atlas.model.instance.AtlasEntity;
import org.apache.atlas.utils.LruCache;
@@ -68,6 +69,7 @@ public class HiveHook extends AtlasHook implements
ExecuteWithHookContext {
public static final String HOOK_HIVE_TABLE_IGNORE_PATTERN
= CONF_PREFIX + "hive_table.ignore.pattern";
public static final String HOOK_HIVE_TABLE_PRUNE_PATTERN
= CONF_PREFIX + "hive_table.prune.pattern";
public static final String HOOK_HIVE_TABLE_CACHE_SIZE
= CONF_PREFIX + "hive_table.cache.size";
+public static final String HOOK_HIVE_IGNORE_DDL_OPERATIONS
= CONF_PREFIX + "hs2.ignore.ddl.operations";
public static final String DEFAULT_HOST_NAME = "localhost";
private static final Map OPERATION_MAP = new
HashMap<>();
@@ -88,7 +90,7 @@ public class HiveHook extends AtlasHook implements
ExecuteWithHookContext {
private static final List ignoreDummyTableName;
private static final String
ignoreValuesTmpTableNamePrefix;
private static final boolean
hiveProcessPopulateDeprecatedAttributes;
-private static HiveHookObjectNamesCache knownObjects = null;
+private static HiveHookObjectNamesCacheknownObjects = null;
private static String hostName;
static {
@@
[atlas] branch master updated: ATLAS-4204: Configure HS2 hook to send lineage-only messages.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 2588540 ATLAS-4204: Configure HS2 hook to send lineage-only messages.
2588540 is described below
commit 2588540ebef16c6fa4890bb15c7841042b13ab3c
Author: Ashutosh Mestry
AuthorDate: Mon Apr 12 09:19:42 2021 -0700
ATLAS-4204: Configure HS2 hook to send lineage-only messages.
---
.../atlas/hive/hook/AtlasHiveHookContext.java | 4 +
.../java/org/apache/atlas/hive/hook/HiveHook.java | 8 +-
.../atlas/hive/hook/events/BaseHiveEvent.java | 2 +
.../atlas/hive/hook/utils/ActiveEntityFilter.java | 49
.../apache/atlas/hive/hook/utils/EntityFilter.java | 26 ++
.../atlas/hive/hook/utils/HiveDDLEntityFilter.java | 258 +++
.../atlas/hive/hook/utils/PassthroughFilter.java | 29 +++
.../hive/hook/utils/ActiveEntityFilterTest.java| 204 +++
.../test/resources/atlas-application.properties| 1 +
.../src/test/resources/json/hs2-alter-view-v2.json | 35 +++
.../src/test/resources/json/hs2-alter-view.json| 268 +++
.../src/test/resources/json/hs2-create-db-v2.json | 30 +++
.../src/test/resources/json/hs2-create-db.json | 73 ++
.../test/resources/json/hs2-create-process-v2.json | 144 +++
.../test/resources/json/hs2-create-process.json| 283 +
.../test/resources/json/hs2-create-table-v2.json | 102
.../src/test/resources/json/hs2-create-table.json | 255 +++
.../src/test/resources/json/hs2-drop-db-v2.json| 4 +
.../src/test/resources/json/hs2-drop-db.json | 10 +
.../src/test/resources/json/hs2-drop-table-v2.json | 1 +
.../src/test/resources/json/hs2-drop-table.json| 50
.../test/resources/json/hs2-table-rename-v2.json | 35 +++
.../src/test/resources/json/hs2-table-rename.json | 276
.../main/java/org/apache/atlas/type/AtlasType.java | 3 +
.../java/org/apache/atlas/utils/AtlasJson.java | 14 +
25 files changed, 2162 insertions(+), 2 deletions(-)
diff --git
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
index 1286471..ab8e183 100644
---
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
+++
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
@@ -246,6 +246,10 @@ public class AtlasHiveHookContext {
}
private void init() {
+if (hiveOperation == null) {
+return;
+}
+
String operation = hiveOperation.getOperationName();
if (knownObjects == null || !isCreateAlterOperation(operation)) {
diff --git
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
index 79e87c7..cf918ef 100644
--- a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
+++ b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
@@ -19,6 +19,7 @@
package org.apache.atlas.hive.hook;
import org.apache.atlas.hive.hook.events.*;
+import org.apache.atlas.hive.hook.utils.ActiveEntityFilter;
import org.apache.atlas.hook.AtlasHook;
import org.apache.atlas.model.instance.AtlasEntity;
import org.apache.atlas.utils.LruCache;
@@ -68,6 +69,7 @@ public class HiveHook extends AtlasHook implements
ExecuteWithHookContext {
public static final String HOOK_HIVE_TABLE_IGNORE_PATTERN
= CONF_PREFIX + "hive_table.ignore.pattern";
public static final String HOOK_HIVE_TABLE_PRUNE_PATTERN
= CONF_PREFIX + "hive_table.prune.pattern";
public static final String HOOK_HIVE_TABLE_CACHE_SIZE
= CONF_PREFIX + "hive_table.cache.size";
+public static final String HOOK_HIVE_IGNORE_DDL_OPERATIONS
= CONF_PREFIX + "hs2.ignore.ddl.operations";
public static final String DEFAULT_HOST_NAME = "localhost";
private static final Map OPERATION_MAP = new
HashMap<>();
@@ -88,7 +90,7 @@ public class HiveHook extends AtlasHook implements
ExecuteWithHookContext {
private static final List ignoreDummyTableName;
private static final String
ignoreValuesTmpTableNamePrefix;
private static final boolean
hiveProcessPopulateDeprecatedAttributes;
-private static HiveHookObjectNamesCache knownObjects = null;
+private static HiveHookObjectNamesCacheknownObjects = null;
private static String hostName;
static {
@@ -158,6 +160
[atlas] branch branch-2.0 updated: ATLAS-4155: NotificationHookConsumer: Fix for Large Message Processing Problem
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new dbdfcd2 ATLAS-4155: NotificationHookConsumer: Fix for Large Message
Processing Problem
dbdfcd2 is described below
commit dbdfcd27f264c6aa0bc6333ece5b992f9f985885
Author: Ashutosh Mestry
AuthorDate: Wed Feb 17 21:17:30 2021 -0800
ATLAS-4155: NotificationHookConsumer: Fix for Large Message Processing
Problem
---
.../org/apache/atlas/kafka/AtlasKafkaConsumer.java | 73 ++
.../atlas/notification/NotificationConsumer.java | 9 ++-
.../AbstractNotificationConsumerTest.java | 6 ++
.../notification/NotificationHookConsumer.java | 50 +++
.../NotificationHookConsumerKafkaTest.java | 5 --
5 files changed, 71 insertions(+), 72 deletions(-)
diff --git
a/notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
b/notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
index c38a504..f7d9668 100644
--- a/notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
+++ b/notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
@@ -20,12 +20,14 @@ package org.apache.atlas.kafka;
import org.apache.atlas.notification.AbstractNotificationConsumer;
import org.apache.atlas.notification.AtlasNotificationMessageDeserializer;
import org.apache.atlas.notification.NotificationInterface;
+import org.apache.commons.collections.MapUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.Map;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
@@ -63,7 +65,40 @@ public class AtlasKafkaConsumer extends
AbstractNotificationConsumer {
@Override
public List> receive(long timeoutMilliSeconds) {
+return receive(this.pollTimeoutMilliSeconds, null);
+}
+
+@Override
+public List>
receiveWithCheckedCommit(Map
lastCommittedPartitionOffset) {
+return receive(this.pollTimeoutMilliSeconds,
lastCommittedPartitionOffset);
+}
+
+
+@Override
+public void commit(TopicPartition partition, long offset) {
+if (!autoCommitEnabled) {
+if (LOG.isDebugEnabled()) {
+LOG.info(" commiting the offset ==>> " + offset);
+}
+kafkaConsumer.commitSync(Collections.singletonMap(partition, new
OffsetAndMetadata(offset)));
+}
+}
+
+@Override
+public void close() {
+if (kafkaConsumer != null) {
+kafkaConsumer.close();
+}
+}
+
+@Override
+public void wakeup() {
+if (kafkaConsumer != null) {
+kafkaConsumer.wakeup();
+}
+}
+private List> receive(long timeoutMilliSeconds,
Map lastCommittedPartitionOffset) {
List> messages = new ArrayList();
ConsumerRecords records = kafkaConsumer != null ?
kafkaConsumer.poll(timeoutMilliSeconds) : null;
@@ -75,13 +110,24 @@ public class AtlasKafkaConsumer extends
AbstractNotificationConsumer {
record.topic(), record.partition(),
record.offset(), record.key(), record.value());
}
+TopicPartition topicPartition = new
TopicPartition(record.topic(), record.partition());
+if (MapUtils.isNotEmpty(lastCommittedPartitionOffset)
+&&
lastCommittedPartitionOffset.containsKey(topicPartition)
+&& record.offset() <
lastCommittedPartitionOffset.get(topicPartition)) {
+
+commit(topicPartition, record.offset());
+LOG.info("Skipping already processed message: topic={},
partition={} offset={}. Last processed offset={}",
+record.topic(), record.partition(),
record.offset(), lastCommittedPartitionOffset.get(topicPartition));
+continue;
+}
+
T message = null;
try {
message =
deserializer.deserialize(record.value().toString());
} catch (OutOfMemoryError excp) {
LOG.error("Ignoring message that failed to deserialize:
topic={}, partition={}, offset={}, key={}, value={}",
- record.topic(), record.partition(),
record.offset(), record.key(), record.value(), excp);
+record.topic(), record.partition(),
record.offset(), record.key(), record.value(), excp);
}
if (message == null) {
@@ -95,29 +141,4 @@ public class AtlasKafkaConsumer extends
AbstractNotificationConsumer {
r
[atlas] branch master updated: ATLAS-4155: NotificationHookConsumer: Fix for Large Message Processing Problem
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new e17ef54 ATLAS-4155: NotificationHookConsumer: Fix for Large Message
Processing Problem
e17ef54 is described below
commit e17ef54d0f701bd5b8521d0ecd2908db32f984f3
Author: Ashutosh Mestry
AuthorDate: Wed Feb 17 21:17:30 2021 -0800
ATLAS-4155: NotificationHookConsumer: Fix for Large Message Processing
Problem
---
.../org/apache/atlas/kafka/AtlasKafkaConsumer.java | 73 ++
.../atlas/notification/NotificationConsumer.java | 9 ++-
.../AbstractNotificationConsumerTest.java | 6 ++
.../notification/NotificationHookConsumer.java | 50 +++
.../NotificationHookConsumerKafkaTest.java | 5 --
5 files changed, 71 insertions(+), 72 deletions(-)
diff --git
a/notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
b/notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
index c38a504..f7d9668 100644
--- a/notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
+++ b/notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
@@ -20,12 +20,14 @@ package org.apache.atlas.kafka;
import org.apache.atlas.notification.AbstractNotificationConsumer;
import org.apache.atlas.notification.AtlasNotificationMessageDeserializer;
import org.apache.atlas.notification.NotificationInterface;
+import org.apache.commons.collections.MapUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.Map;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
@@ -63,7 +65,40 @@ public class AtlasKafkaConsumer extends
AbstractNotificationConsumer {
@Override
public List> receive(long timeoutMilliSeconds) {
+return receive(this.pollTimeoutMilliSeconds, null);
+}
+
+@Override
+public List>
receiveWithCheckedCommit(Map
lastCommittedPartitionOffset) {
+return receive(this.pollTimeoutMilliSeconds,
lastCommittedPartitionOffset);
+}
+
+
+@Override
+public void commit(TopicPartition partition, long offset) {
+if (!autoCommitEnabled) {
+if (LOG.isDebugEnabled()) {
+LOG.info(" commiting the offset ==>> " + offset);
+}
+kafkaConsumer.commitSync(Collections.singletonMap(partition, new
OffsetAndMetadata(offset)));
+}
+}
+
+@Override
+public void close() {
+if (kafkaConsumer != null) {
+kafkaConsumer.close();
+}
+}
+
+@Override
+public void wakeup() {
+if (kafkaConsumer != null) {
+kafkaConsumer.wakeup();
+}
+}
+private List> receive(long timeoutMilliSeconds,
Map lastCommittedPartitionOffset) {
List> messages = new ArrayList();
ConsumerRecords records = kafkaConsumer != null ?
kafkaConsumer.poll(timeoutMilliSeconds) : null;
@@ -75,13 +110,24 @@ public class AtlasKafkaConsumer extends
AbstractNotificationConsumer {
record.topic(), record.partition(),
record.offset(), record.key(), record.value());
}
+TopicPartition topicPartition = new
TopicPartition(record.topic(), record.partition());
+if (MapUtils.isNotEmpty(lastCommittedPartitionOffset)
+&&
lastCommittedPartitionOffset.containsKey(topicPartition)
+&& record.offset() <
lastCommittedPartitionOffset.get(topicPartition)) {
+
+commit(topicPartition, record.offset());
+LOG.info("Skipping already processed message: topic={},
partition={} offset={}. Last processed offset={}",
+record.topic(), record.partition(),
record.offset(), lastCommittedPartitionOffset.get(topicPartition));
+continue;
+}
+
T message = null;
try {
message =
deserializer.deserialize(record.value().toString());
} catch (OutOfMemoryError excp) {
LOG.error("Ignoring message that failed to deserialize:
topic={}, partition={}, offset={}, key={}, value={}",
- record.topic(), record.partition(),
record.offset(), record.key(), record.value(), excp);
+record.topic(), record.partition(),
record.offset(), record.key(), record.value(), excp);
}
if (message == null) {
@@ -95,29 +141,4 @@ public class AtlasKafkaConsumer extends
AbstractNotificationConsumer {
r
[atlas] branch branch-2.0 updated: ATLAS-4109: Advanced Search: Glossary: More Efficient Structure
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 47a2bb8 ATLAS-4109: Advanced Search: Glossary: More Efficient
Structure
47a2bb8 is described below
commit 47a2bb86fdb2d7207feee04fd9d0abb82fa1b46c
Author: Ashutosh Mestry
AuthorDate: Wed Feb 3 13:44:31 2021 -0800
ATLAS-4109: Advanced Search: Glossary: More Efficient Structure
---
.../java/org/apache/atlas/query/GremlinClause.java | 3 ++-
.../executors/GremlinClauseToTraversalTranslator.java | 10 --
.../apache/atlas/query/GremlinQueryComposerTest.java | 18 +++---
3 files changed, 13 insertions(+), 18 deletions(-)
diff --git a/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
b/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
index b7e5330..7014b4a 100644
--- a/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
+++ b/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
@@ -69,7 +69,8 @@ public enum GremlinClause {
// idx of the tuple field to be sorted on
INLINE_TUPLE_SORT_ASC(".sort{a,b -> a[%s] <=> b[%s]}"),
INLINE_TUPLE_SORT_DESC(".sort{a,b -> b[%s] <=> a[%s]}"),
-
TERM("and(__.in('r:AtlasGlossarySemanticAssignment').has('AtlasGlossaryTerm.%s',
eq('%s')).dedup())"),
+
TERM("where(in('r:AtlasGlossarySemanticAssignment').has('AtlasGlossaryTerm.%s',
'%s'))"),
+
V("V()"),
VALUE_MAP("valueMap(%s)");
diff --git
a/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
b/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
index 156a65c..26942c7 100644
---
a/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
+++
b/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
@@ -248,12 +248,10 @@ public class GremlinClauseToTraversalTranslator {
case TERM:
String term = String.format("AtlasGlossaryTerm.%s",
values[0]);
-
-traversal.and(
-traversal.startAnonymousTraversal()
-
.in(org.apache.atlas.repository.Constants.TERM_ASSIGNMENT_LABEL)
- .has(term, P.eq(values[1]))
- .dedup()
+traversal.where(
+ traversal.startAnonymousTraversal()
+
.in(org.apache.atlas.repository.Constants.TERM_ASSIGNMENT_LABEL)
+.has(term, P.eq(values[1]))
);
break;
diff --git
a/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
b/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
index 3594e2f..3a2d66d 100644
---
a/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
+++
b/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
@@ -334,21 +334,17 @@ public class GremlinQueryComposerTest {
@Test
public void glossaryTermQueries() {
-verify("Table hasTerm sales", "g.V().has('__typeName', 'Table')." +
-"and(__.in('r:AtlasGlossarySemanticAssignment')." +
-"has('AtlasGlossaryTerm.name', eq('sales')).dedup())." +
-"dedup().limit(25).toList()");
+verify("Table hasTerm sales", "g.V().has('__typeName', 'Table')" +
+
".where(in('r:AtlasGlossarySemanticAssignment').has('AtlasGlossaryTerm.name',
'sales'))" +
+".dedup().limit(25).toList()");
verify("Table hasTerm \"sales@glossary\"", "g.V().has('__typeName',
'Table')." +
-"and(__.in('r:AtlasGlossarySemanticAssignment')." +
-"has('AtlasGlossaryTerm.qualifiedName',
eq('sales@glossary')).dedup())." +
-"dedup().limit(25).toList()");
+
"where(in('r:AtlasGlossarySemanticAssignment').has('AtlasGlossaryTerm.qualifiedName',
'sales@glossary'))" +
+".dedup().limit(25).toList()");
verify("Table hasTerm \"sales@glossary\" and owner = \"fetl\"",
"g.V().has('__typeName', 'Table')" +
".and(" +
-"__.and(" +
-
"__.in('r:AtlasGlossarySemanticAssignment'
[atlas] branch master updated: ATLAS-4109: Advanced Search: Glossary: More Efficient Structure
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new ed2926e ATLAS-4109: Advanced Search: Glossary: More Efficient
Structure
ed2926e is described below
commit ed2926ece4d1ba7a3fa773f527853253bd3352ba
Author: Ashutosh Mestry
AuthorDate: Wed Feb 3 13:44:31 2021 -0800
ATLAS-4109: Advanced Search: Glossary: More Efficient Structure
---
.../java/org/apache/atlas/query/GremlinClause.java | 3 ++-
.../executors/GremlinClauseToTraversalTranslator.java | 10 --
.../apache/atlas/query/GremlinQueryComposerTest.java | 18 +++---
3 files changed, 13 insertions(+), 18 deletions(-)
diff --git a/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
b/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
index b7e5330..7014b4a 100644
--- a/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
+++ b/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
@@ -69,7 +69,8 @@ public enum GremlinClause {
// idx of the tuple field to be sorted on
INLINE_TUPLE_SORT_ASC(".sort{a,b -> a[%s] <=> b[%s]}"),
INLINE_TUPLE_SORT_DESC(".sort{a,b -> b[%s] <=> a[%s]}"),
-
TERM("and(__.in('r:AtlasGlossarySemanticAssignment').has('AtlasGlossaryTerm.%s',
eq('%s')).dedup())"),
+
TERM("where(in('r:AtlasGlossarySemanticAssignment').has('AtlasGlossaryTerm.%s',
'%s'))"),
+
V("V()"),
VALUE_MAP("valueMap(%s)");
diff --git
a/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
b/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
index 156a65c..26942c7 100644
---
a/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
+++
b/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
@@ -248,12 +248,10 @@ public class GremlinClauseToTraversalTranslator {
case TERM:
String term = String.format("AtlasGlossaryTerm.%s",
values[0]);
-
-traversal.and(
-traversal.startAnonymousTraversal()
-
.in(org.apache.atlas.repository.Constants.TERM_ASSIGNMENT_LABEL)
- .has(term, P.eq(values[1]))
- .dedup()
+traversal.where(
+ traversal.startAnonymousTraversal()
+
.in(org.apache.atlas.repository.Constants.TERM_ASSIGNMENT_LABEL)
+.has(term, P.eq(values[1]))
);
break;
diff --git
a/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
b/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
index 3594e2f..3a2d66d 100644
---
a/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
+++
b/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
@@ -334,21 +334,17 @@ public class GremlinQueryComposerTest {
@Test
public void glossaryTermQueries() {
-verify("Table hasTerm sales", "g.V().has('__typeName', 'Table')." +
-"and(__.in('r:AtlasGlossarySemanticAssignment')." +
-"has('AtlasGlossaryTerm.name', eq('sales')).dedup())." +
-"dedup().limit(25).toList()");
+verify("Table hasTerm sales", "g.V().has('__typeName', 'Table')" +
+
".where(in('r:AtlasGlossarySemanticAssignment').has('AtlasGlossaryTerm.name',
'sales'))" +
+".dedup().limit(25).toList()");
verify("Table hasTerm \"sales@glossary\"", "g.V().has('__typeName',
'Table')." +
-"and(__.in('r:AtlasGlossarySemanticAssignment')." +
-"has('AtlasGlossaryTerm.qualifiedName',
eq('sales@glossary')).dedup())." +
-"dedup().limit(25).toList()");
+
"where(in('r:AtlasGlossarySemanticAssignment').has('AtlasGlossaryTerm.qualifiedName',
'sales@glossary'))" +
+".dedup().limit(25).toList()");
verify("Table hasTerm \"sales@glossary\" and owner = \"fetl\"",
"g.V().has('__typeName', 'Table')" +
".and(" +
-"__.and(" +
-
"__.in('r:AtlasGlossarySemanticAssignment'
[atlas] branch branch-2.0 updated: ATLAS-4151: FixedBufferList: Change Log Level to Debug
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 73c9a27 ATLAS-4151: FixedBufferList: Change Log Level to Debug
73c9a27 is described below
commit 73c9a2700a39fb8a3a749b4a3c046fb3caf7b035
Author: Ashutosh Mestry
AuthorDate: Tue Feb 9 09:08:54 2021 -0800
ATLAS-4151: FixedBufferList: Change Log Level to Debug
---
intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
b/intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
index cf6ec02..d407151 100644
--- a/intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
+++ b/intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
@@ -70,7 +70,9 @@ public class FixedBufferList {
this.buffer.ensureCapacity(newCapacity);
instantiateItems(oldCapacity, newCapacity);
-LOG.info("FixedBufferList: Requested: {} From: {} To:{}",
requestedCapacity, oldCapacity, newCapacity);
+if (LOG.isDebugEnabled()) {
+LOG.debug("FixedBufferList: Requested: {} From: {} To:{}",
requestedCapacity, oldCapacity, newCapacity);
+}
}
private void instantiateItems(int startIndex, int maxSize) {
[atlas] branch master updated: ATLAS-4151: FixedBufferList: Change Log Level to Debug
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 2105d5a ATLAS-4151: FixedBufferList: Change Log Level to Debug
2105d5a is described below
commit 2105d5a2d15502e00eda8b1c657952f5ee96cb94
Author: Ashutosh Mestry
AuthorDate: Tue Feb 9 09:08:54 2021 -0800
ATLAS-4151: FixedBufferList: Change Log Level to Debug
---
intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
b/intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
index cf6ec02..d407151 100644
--- a/intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
+++ b/intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
@@ -70,7 +70,9 @@ public class FixedBufferList {
this.buffer.ensureCapacity(newCapacity);
instantiateItems(oldCapacity, newCapacity);
-LOG.info("FixedBufferList: Requested: {} From: {} To:{}",
requestedCapacity, oldCapacity, newCapacity);
+if (LOG.isDebugEnabled()) {
+LOG.debug("FixedBufferList: Requested: {} From: {} To:{}",
requestedCapacity, oldCapacity, newCapacity);
+}
}
private void instantiateItems(int startIndex, int maxSize) {
[atlas] 01/02: ATLAS-4137 Fix for broken V1 entity audit API
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit baf31cf70ec2bc1b3e36ae09eaebf8d13830
Author: Deep Singh
AuthorDate: Thu Feb 4 12:13:04 2021 -0600
ATLAS-4137 Fix for broken V1 entity audit API
Signed-off-by: Ashutosh Mestry
---
.../org/apache/atlas/repository/converters/AtlasInstanceConverter.java | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
b/repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
index 6fc0c65..5c08874 100644
---
a/repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
+++
b/repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
@@ -384,6 +384,8 @@ public class AtlasInstanceConverter {
case ENTITY_CREATE:
return EntityAuditEvent.EntityAuditAction.ENTITY_CREATE;
case ENTITY_UPDATE:
+case BUSINESS_ATTRIBUTE_UPDATE:
+case CUSTOM_ATTRIBUTE_UPDATE:
return EntityAuditEvent.EntityAuditAction.ENTITY_UPDATE;
case ENTITY_DELETE:
return EntityAuditEvent.EntityAuditAction.ENTITY_DELETE;
[atlas] branch branch-2.0 updated (045c849 -> 4193650)
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a change to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git.
from 045c849 Atlas-4166: UI should not allow the user to create a glossary
with blank(" ") name
new baf31cf ATLAS-4137 Fix for broken V1 entity audit API
new 4193650 ATLAS-4136: Export Service: NPE if Options Explicitly set to
NULL
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../converters/AtlasInstanceConverter.java | 2 ++
.../apache/atlas/web/resources/AdminResource.java | 25 +-
2 files changed, 17 insertions(+), 10 deletions(-)
[atlas] 02/02: ATLAS-4136: Export Service: NPE if Options Explicitly set to NULL
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 41936504d59f3c5cbcda28aeeba8b6d0c489509c
Author: Ashutosh Mestry
AuthorDate: Thu Feb 4 09:58:57 2021 -0800
ATLAS-4136: Export Service: NPE if Options Explicitly set to NULL
---
.../apache/atlas/web/resources/AdminResource.java | 25 +-
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git
a/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
b/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
index b20b404..e8fc111 100755
--- a/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
+++ b/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
@@ -390,7 +390,7 @@ public class AdminResource {
AtlasAuthorizationUtils.verifyAccess(new
AtlasAdminAccessRequest(AtlasPrivilege.ADMIN_EXPORT), "export");
-boolean preventMultipleRequests = request != null
+boolean preventMultipleRequests = request != null &&
request.getOptions() != null
&&
!(request.getOptions().containsKey(AtlasExportRequest.OPTION_SKIP_LINEAGE)
||
request.getOptions().containsKey(AtlasExportRequest.OPTION_KEY_REPLICATED_TO));
if (preventMultipleRequests) {
@@ -455,7 +455,8 @@ public class AdminResource {
try {
AtlasImportRequest request = AtlasType.fromJson(jsonData,
AtlasImportRequest.class);
-preventMultipleRequests = request != null &&
!request.getOptions().containsKey(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM);
+preventMultipleRequests = request != null && request.getOptions()
!= null
+&&
!request.getOptions().containsKey(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM);
if (preventMultipleRequests) {
acquireExportImportLock("import");
}
@@ -537,7 +538,8 @@ public class AdminResource {
try {
AtlasImportRequest request = AtlasType.fromJson(jsonData,
AtlasImportRequest.class);
-preventMultipleRequests = request != null &&
request.getOptions().containsKey(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM);
+preventMultipleRequests = request != null && request.getOptions()
!= null
+&&
request.getOptions().containsKey(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM);
if (preventMultipleRequests) {
acquireExportImportLock("importFile");
@@ -783,16 +785,19 @@ public class AdminResource {
}
private void addToExportOperationAudits(boolean isSuccessful,
AtlasExportResult result) throws AtlasBaseException {
-if (isSuccessful &&
CollectionUtils.isNotEmpty(result.getRequest().getItemsToExport())) {
+if (!isSuccessful
+||
CollectionUtils.isEmpty(result.getRequest().getItemsToExport())
+|| result.getRequest().getOptions() == null) {
+return;
+}
-Map optionMap = result.getRequest().getOptions();
-optionMap.put(OPERATION_STATUS,
result.getOperationStatus().name());
-String params = AtlasJson.toJson(optionMap);
+Map optionMap = result.getRequest().getOptions();
+optionMap.put(OPERATION_STATUS, result.getOperationStatus().name());
+String params = AtlasJson.toJson(optionMap);
-List objectIds =
result.getRequest().getItemsToExport();
+List objectIds = result.getRequest().getItemsToExport();
-auditImportExportOperations(objectIds, AuditOperation.EXPORT,
params);
-}
+auditImportExportOperations(objectIds, AuditOperation.EXPORT, params);
}
private void auditImportExportOperations(List objectIds,
AuditOperation auditOperation, String params) throws AtlasBaseException {
[atlas] branch master updated (7a3f512 -> 960f948)
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git.
from 7a3f512 Atlas-4166: UI should not allow the user to create a glossary
with blank(" ") name
new 15cd037 ATLAS-4136: Export Service: NPE if Options Explicitly set to
NULL
new 960f948 ATLAS-4137 Fix for broken V1 entity audit API
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../converters/AtlasInstanceConverter.java | 2 ++
.../apache/atlas/web/resources/AdminResource.java | 25 +-
2 files changed, 17 insertions(+), 10 deletions(-)
[atlas] 01/02: ATLAS-4136: Export Service: NPE if Options Explicitly set to NULL
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 15cd0378ef54720349f28ef3f2624f93b4f5b30e
Author: Ashutosh Mestry
AuthorDate: Thu Feb 4 09:58:57 2021 -0800
ATLAS-4136: Export Service: NPE if Options Explicitly set to NULL
---
.../apache/atlas/web/resources/AdminResource.java | 25 +-
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git
a/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
b/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
index b20b404..e8fc111 100755
--- a/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
+++ b/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
@@ -390,7 +390,7 @@ public class AdminResource {
AtlasAuthorizationUtils.verifyAccess(new
AtlasAdminAccessRequest(AtlasPrivilege.ADMIN_EXPORT), "export");
-boolean preventMultipleRequests = request != null
+boolean preventMultipleRequests = request != null &&
request.getOptions() != null
&&
!(request.getOptions().containsKey(AtlasExportRequest.OPTION_SKIP_LINEAGE)
||
request.getOptions().containsKey(AtlasExportRequest.OPTION_KEY_REPLICATED_TO));
if (preventMultipleRequests) {
@@ -455,7 +455,8 @@ public class AdminResource {
try {
AtlasImportRequest request = AtlasType.fromJson(jsonData,
AtlasImportRequest.class);
-preventMultipleRequests = request != null &&
!request.getOptions().containsKey(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM);
+preventMultipleRequests = request != null && request.getOptions()
!= null
+&&
!request.getOptions().containsKey(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM);
if (preventMultipleRequests) {
acquireExportImportLock("import");
}
@@ -537,7 +538,8 @@ public class AdminResource {
try {
AtlasImportRequest request = AtlasType.fromJson(jsonData,
AtlasImportRequest.class);
-preventMultipleRequests = request != null &&
request.getOptions().containsKey(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM);
+preventMultipleRequests = request != null && request.getOptions()
!= null
+&&
request.getOptions().containsKey(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM);
if (preventMultipleRequests) {
acquireExportImportLock("importFile");
@@ -783,16 +785,19 @@ public class AdminResource {
}
private void addToExportOperationAudits(boolean isSuccessful,
AtlasExportResult result) throws AtlasBaseException {
-if (isSuccessful &&
CollectionUtils.isNotEmpty(result.getRequest().getItemsToExport())) {
+if (!isSuccessful
+||
CollectionUtils.isEmpty(result.getRequest().getItemsToExport())
+|| result.getRequest().getOptions() == null) {
+return;
+}
-Map optionMap = result.getRequest().getOptions();
-optionMap.put(OPERATION_STATUS,
result.getOperationStatus().name());
-String params = AtlasJson.toJson(optionMap);
+Map optionMap = result.getRequest().getOptions();
+optionMap.put(OPERATION_STATUS, result.getOperationStatus().name());
+String params = AtlasJson.toJson(optionMap);
-List objectIds =
result.getRequest().getItemsToExport();
+List objectIds = result.getRequest().getItemsToExport();
-auditImportExportOperations(objectIds, AuditOperation.EXPORT,
params);
-}
+auditImportExportOperations(objectIds, AuditOperation.EXPORT, params);
}
private void auditImportExportOperations(List objectIds,
AuditOperation auditOperation, String params) throws AtlasBaseException {
[atlas] 02/02: ATLAS-4137 Fix for broken V1 entity audit API
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 960f9481e6914e6591687c13e084a90fec1c2d02
Author: Deep Singh
AuthorDate: Thu Feb 4 12:13:04 2021 -0600
ATLAS-4137 Fix for broken V1 entity audit API
Signed-off-by: Ashutosh Mestry
---
.../org/apache/atlas/repository/converters/AtlasInstanceConverter.java | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
b/repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
index 6fc0c65..5c08874 100644
---
a/repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
+++
b/repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
@@ -384,6 +384,8 @@ public class AtlasInstanceConverter {
case ENTITY_CREATE:
return EntityAuditEvent.EntityAuditAction.ENTITY_CREATE;
case ENTITY_UPDATE:
+case BUSINESS_ATTRIBUTE_UPDATE:
+case CUSTOM_ATTRIBUTE_UPDATE:
return EntityAuditEvent.EntityAuditAction.ENTITY_UPDATE;
case ENTITY_DELETE:
return EntityAuditEvent.EntityAuditAction.ENTITY_DELETE;
[atlas] branch branch-2.0 updated: ATLAS-4122: Advanced Search: Literals with double-quotes.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 857ffa2 ATLAS-4122: Advanced Search: Literals with double-quotes.
857ffa2 is described below
commit 857ffa2df5d08b5c336ff641fa5c10b9e589be70
Author: Ashutosh Mestry
AuthorDate: Tue Feb 2 20:59:07 2021 -0800
ATLAS-4122: Advanced Search: Literals with double-quotes.
---
.../query/executors/GremlinClauseToTraversalTranslator.java | 5 +++--
.../src/test/java/org/apache/atlas/query/DSLQueriesTest.java | 9 ++---
.../java/org/apache/atlas/query/GremlinQueryComposerTest.java| 8
3 files changed, 17 insertions(+), 5 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
b/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
index a7b05fd..156a65c 100644
---
a/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
+++
b/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
@@ -48,6 +48,8 @@ public class GremlinClauseToTraversalTranslator {
private static final String EDGE_NAME_CLASSIFIED_AS =
"classifiedAs";
private static final String EDGE_NAME_TRAIT_NAMES=
"__traitNames";
private static final String EDGE_NAME_PROPAGATED_TRAIT_NAMES =
"__propagatedTraitNames";
+private static final String[] STR_TOKEN_SEARCH = new
String[]{"[", "]", "'", "\""};
+private static final String[] STR_TOKEN_REPLACE = new
String[]{"", "", "", ""};
private final AtlasGraph graph;
@@ -293,8 +295,7 @@ public class GremlinClauseToTraversalTranslator {
}
private String[] csvToArray(String strRhs) {
-String csvRow = StringUtils.replaceEach(strRhs, new String[]{"[",
"]", "'"}, new String[]{"", "", ""});
-
+String csvRow = StringUtils.replaceEach(strRhs, STR_TOKEN_SEARCH,
STR_TOKEN_REPLACE);
return csvRow.split(",");
}
diff --git
a/repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
b/repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
index 359cb31..0c1dd59 100644
--- a/repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
+++ b/repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
@@ -226,7 +226,8 @@ public class DSLQueriesTest extends BasicTestSetup {
{"hive_table hasTerm \"modernTrade@salesGlossary\" and
hive_table isA Dimension",1, new ListValidator( "time_dim")},
{"hive_table hasTerm \"modernTrade@salesGlossary\" and db.name
= \"Sales\" or (hive_table.name = \"sales_fact_monthly_mv\")", 2, new
ListValidator("sales_fact_monthly_mv", "time_dim")},
{"hive_table where hive_table hasTerm
\"modernTrade@salesGlossary\"", 2, new ListValidator("logging_fact_monthly_mv",
"time_dim")},
-{"hive_table where (name = \"product_dim\" and hive_table
hasTerm \"ecommerce@salesGlossary\")", 1, new ListValidator("product_dim")}
+{"hive_table where (name = \"product_dim\" and hive_table
hasTerm \"ecommerce@salesGlossary\")", 1, new ListValidator("product_dim")},
+{"hive_table where (name = 'product_dim' and hive_table
hasTerm 'ecommerce@salesGlossary')", 1, new ListValidator("product_dim")}
};
}
@@ -283,14 +284,16 @@ public class DSLQueriesTest extends BasicTestSetup {
{"DataSet where name='sales_fact'", 1},
{"Asset where name='sales_fact'", 1},
{"hive_db _NOT_CLASSIFIED", 3},
-{"_CLASSIFIED", 23}
+{"_CLASSIFIED", 23},
+{"hive_db where name = [\"Reporting\",\"Sales\"]", 2},
+{"hive_db where name = ['Reporting', 'Sales']", 2},
};
}
@Test(dataProvider = "basicProvider")
public void basic(String query, int expected) throws AtlasBaseException {
queryAssert(query, expected, DEFAULT_LIMIT, 0);
-//queryAssert(query.replace("where", " "), expected, DEFAULT_LIMIT, 0);
+queryAssert(query.replace("where"
[atlas] branch master updated: ATLAS-4122: Advanced Search: Literals with double-quotes.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new efcc31e ATLAS-4122: Advanced Search: Literals with double-quotes.
efcc31e is described below
commit efcc31e8dcd5038eaa964386c3e123fb90f1b28b
Author: Ashutosh Mestry
AuthorDate: Tue Feb 2 20:59:07 2021 -0800
ATLAS-4122: Advanced Search: Literals with double-quotes.
---
.../query/executors/GremlinClauseToTraversalTranslator.java | 5 +++--
.../src/test/java/org/apache/atlas/query/DSLQueriesTest.java | 9 ++---
.../java/org/apache/atlas/query/GremlinQueryComposerTest.java| 8
3 files changed, 17 insertions(+), 5 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
b/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
index a7b05fd..156a65c 100644
---
a/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
+++
b/repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
@@ -48,6 +48,8 @@ public class GremlinClauseToTraversalTranslator {
private static final String EDGE_NAME_CLASSIFIED_AS =
"classifiedAs";
private static final String EDGE_NAME_TRAIT_NAMES=
"__traitNames";
private static final String EDGE_NAME_PROPAGATED_TRAIT_NAMES =
"__propagatedTraitNames";
+private static final String[] STR_TOKEN_SEARCH = new
String[]{"[", "]", "'", "\""};
+private static final String[] STR_TOKEN_REPLACE = new
String[]{"", "", "", ""};
private final AtlasGraph graph;
@@ -293,8 +295,7 @@ public class GremlinClauseToTraversalTranslator {
}
private String[] csvToArray(String strRhs) {
-String csvRow = StringUtils.replaceEach(strRhs, new String[]{"[",
"]", "'"}, new String[]{"", "", ""});
-
+String csvRow = StringUtils.replaceEach(strRhs, STR_TOKEN_SEARCH,
STR_TOKEN_REPLACE);
return csvRow.split(",");
}
diff --git
a/repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
b/repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
index 359cb31..0c1dd59 100644
--- a/repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
+++ b/repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
@@ -226,7 +226,8 @@ public class DSLQueriesTest extends BasicTestSetup {
{"hive_table hasTerm \"modernTrade@salesGlossary\" and
hive_table isA Dimension",1, new ListValidator( "time_dim")},
{"hive_table hasTerm \"modernTrade@salesGlossary\" and db.name
= \"Sales\" or (hive_table.name = \"sales_fact_monthly_mv\")", 2, new
ListValidator("sales_fact_monthly_mv", "time_dim")},
{"hive_table where hive_table hasTerm
\"modernTrade@salesGlossary\"", 2, new ListValidator("logging_fact_monthly_mv",
"time_dim")},
-{"hive_table where (name = \"product_dim\" and hive_table
hasTerm \"ecommerce@salesGlossary\")", 1, new ListValidator("product_dim")}
+{"hive_table where (name = \"product_dim\" and hive_table
hasTerm \"ecommerce@salesGlossary\")", 1, new ListValidator("product_dim")},
+{"hive_table where (name = 'product_dim' and hive_table
hasTerm 'ecommerce@salesGlossary')", 1, new ListValidator("product_dim")}
};
}
@@ -283,14 +284,16 @@ public class DSLQueriesTest extends BasicTestSetup {
{"DataSet where name='sales_fact'", 1},
{"Asset where name='sales_fact'", 1},
{"hive_db _NOT_CLASSIFIED", 3},
-{"_CLASSIFIED", 23}
+{"_CLASSIFIED", 23},
+{"hive_db where name = [\"Reporting\",\"Sales\"]", 2},
+{"hive_db where name = ['Reporting', 'Sales']", 2},
};
}
@Test(dataProvider = "basicProvider")
public void basic(String query, int expected) throws AtlasBaseException {
queryAssert(query, expected, DEFAULT_LIMIT, 0);
-//queryAssert(query.replace("where", " "), expected, DEFAULT_LIMIT, 0);
+queryAssert(query.replace("where"
[atlas] branch branch-2.0 updated: ATLAS-4015: Add Re-indexing as JAVA_PATCH. Part 2
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new e8dcde5 ATLAS-4015: Add Re-indexing as JAVA_PATCH. Part 2
e8dcde5 is described below
commit e8dcde528deb0e9d5a9edbc28cb3ccb6f87cba0f
Author: Ashutosh Mestry
AuthorDate: Fri Nov 13 10:15:23 2020 -0800
ATLAS-4015: Add Re-indexing as JAVA_PATCH. Part 2
---
.../src/main/java/org/apache/atlas/AtlasConfiguration.java | 2 +-
.../org/apache/atlas/repository/patches/ReIndexPatch.java | 14 --
2 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
index 2d3dfdf..ea9f26d 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
@@ -74,7 +74,7 @@ public enum AtlasConfiguration {
HTTP_HEADER_SERVER_VALUE("atlas.http.header.server.value","Apache Atlas"),
STORAGE_CONSISTENCY_LOCK_ENABLED("atlas.graph.storage.consistency-lock.enabled",
true),
-REINDEX_PATCH_ENABLED("atlas.patch.reindex.enabled", false);
+REBUILD_INDEX("atlas.rebuild.index", false);
private static final Configuration APPLICATION_PROPERTIES;
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/ReIndexPatch.java
b/repository/src/main/java/org/apache/atlas/repository/patches/ReIndexPatch.java
index a47a2cc..38d230f 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/ReIndexPatch.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/ReIndexPatch.java
@@ -23,6 +23,7 @@ import org.apache.atlas.pc.WorkItemBuilder;
import org.apache.atlas.pc.WorkItemConsumer;
import org.apache.atlas.pc.WorkItemManager;
import org.apache.atlas.repository.Constants;
+import org.apache.atlas.repository.graphdb.AtlasEdge;
import org.apache.atlas.repository.graphdb.AtlasElement;
import org.apache.atlas.repository.graphdb.AtlasGraph;
import org.apache.atlas.repository.graphdb.AtlasVertex;
@@ -53,7 +54,8 @@ public class ReIndexPatch extends AtlasPatchHandler {
@Override
public void apply() throws AtlasBaseException {
-if (AtlasConfiguration.REINDEX_PATCH_ENABLED.getBoolean() == false) {
+if (AtlasConfiguration.REBUILD_INDEX.getBoolean() == false) {
+LOG.info("ReIndexPatch: Skipped, since not enabled!");
return;
}
@@ -113,8 +115,8 @@ public class ReIndexPatch extends AtlasPatchHandler {
}
private static void edges(WorkItemManager manager, AtlasGraph graph) {
-Iterable iterable = graph.getEdges();
-for (Iterator iter = iterable.iterator(); iter.hasNext();
) {
+Iterable iterable = graph.getEdges();
+for (Iterator iter = iterable.iterator();
iter.hasNext(); ) {
manager.checkProduce(iter.next());
}
}
@@ -144,9 +146,9 @@ public class ReIndexPatch extends AtlasPatchHandler {
}
private static class ReindexConsumer extends
WorkItemConsumer {
-private List list = new ArrayList();
-private AtlasGraph graph;
-private String[] indexNames;
+private final List list = new ArrayList();
+private final String[] indexNames;
+private final AtlasGraph graph;
private final AtomicLong counter;
public ReindexConsumer(BlockingQueue queue, AtlasGraph graph, String[]
indexNames) {
[atlas] branch branch-2.0 updated (5aec0b8 -> 2a42e78)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git. from 5aec0b8 ATLAS-4058: UI: Fix Entity detail page Hive Column Position with 0 is set to NA. new 9f20077 ATLAS-4076: Propagated-tag-addition entries were appearing multiple times on derived table audit new 2a42e78 ATLAS-4067: Updates in Custom Attributes and Business Attributes must be captured separately in entity audits. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: dashboardv2/public/js/utils/Enums.js | 1 + dashboardv3/public/js/utils/Enums.js | 1 + .../apache/atlas/exception/AtlasBaseException.java | 1 - .../atlas/model/audit/EntityAuditEventV2.java | 4 ++- .../repository/audit/EntityAuditListenerV2.java| 13 - .../store/graph/v2/AtlasEntityChangeNotifier.java | 2 ++ .../store/graph/v2/AtlasEntityStoreV2.java | 18 .../main/java/org/apache/atlas/RequestContext.java | 32 ++ 8 files changed, 64 insertions(+), 8 deletions(-)
[atlas] 02/02: ATLAS-4067: Updates in Custom Attributes and Business Attributes must be captured separately in entity audits.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 2a42e7826e715f8b05d27c7d65d9b2c753e12b57
Author: Deep Singh
AuthorDate: Mon Dec 14 12:40:08 2020 -0600
ATLAS-4067: Updates in Custom Attributes and Business Attributes must be
captured separately in entity audits.
Signed-off-by: Ashutosh Mestry
---
dashboardv2/public/js/utils/Enums.js | 1 +
dashboardv3/public/js/utils/Enums.js | 1 +
.../apache/atlas/exception/AtlasBaseException.java | 1 -
.../atlas/model/audit/EntityAuditEventV2.java | 4 +++-
.../repository/audit/EntityAuditListenerV2.java| 13 +++-
.../store/graph/v2/AtlasEntityStoreV2.java | 18 +++-
.../main/java/org/apache/atlas/RequestContext.java | 24 ++
7 files changed, 54 insertions(+), 8 deletions(-)
diff --git a/dashboardv2/public/js/utils/Enums.js
b/dashboardv2/public/js/utils/Enums.js
index ebfceba..d2e476d 100644
--- a/dashboardv2/public/js/utils/Enums.js
+++ b/dashboardv2/public/js/utils/Enums.js
@@ -42,6 +42,7 @@ define(["require", "backbone"], function(require) {
BUSINESS_ATTRIBUTE_ADD: "Business Attribute(s) Added",
BUSINESS_ATTRIBUTE_UPDATE: "Business Attribute(s) Updated",
BUSINESS_ATTRIBUTE_DELETE: "Business Attribute(s) Deleted",
+CUSTOM_ATTRIBUTE_UPDATE: "User-defined Attribute(s) Updated",
TYPE_DEF_UPDATE: "Type Updated",
TYPE_DEF_CREATE: "Type Created",
TYPE_DEF_DELETE: "Type Deleted",
diff --git a/dashboardv3/public/js/utils/Enums.js
b/dashboardv3/public/js/utils/Enums.js
index ebfceba..d2e476d 100644
--- a/dashboardv3/public/js/utils/Enums.js
+++ b/dashboardv3/public/js/utils/Enums.js
@@ -42,6 +42,7 @@ define(["require", "backbone"], function(require) {
BUSINESS_ATTRIBUTE_ADD: "Business Attribute(s) Added",
BUSINESS_ATTRIBUTE_UPDATE: "Business Attribute(s) Updated",
BUSINESS_ATTRIBUTE_DELETE: "Business Attribute(s) Deleted",
+CUSTOM_ATTRIBUTE_UPDATE: "User-defined Attribute(s) Updated",
TYPE_DEF_UPDATE: "Type Updated",
TYPE_DEF_CREATE: "Type Created",
TYPE_DEF_DELETE: "Type Deleted",
diff --git
a/intg/src/main/java/org/apache/atlas/exception/AtlasBaseException.java
b/intg/src/main/java/org/apache/atlas/exception/AtlasBaseException.java
index 5538702..88a26f4 100644
--- a/intg/src/main/java/org/apache/atlas/exception/AtlasBaseException.java
+++ b/intg/src/main/java/org/apache/atlas/exception/AtlasBaseException.java
@@ -19,7 +19,6 @@ package org.apache.atlas.exception;
import org.apache.atlas.AtlasErrorCode;
-import javax.ws.rs.core.Response;
import java.util.List;
/**
diff --git
a/intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
b/intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
index c37e282..083acac 100644
--- a/intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
+++ b/intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
@@ -54,7 +54,7 @@ public class EntityAuditEventV2 implements Serializable,
Clearable {
CLASSIFICATION_ADD, CLASSIFICATION_DELETE, CLASSIFICATION_UPDATE,
PROPAGATED_CLASSIFICATION_ADD, PROPAGATED_CLASSIFICATION_DELETE,
PROPAGATED_CLASSIFICATION_UPDATE,
TERM_ADD, TERM_DELETE, LABEL_ADD, LABEL_DELETE, ENTITY_PURGE,
-BUSINESS_ATTRIBUTE_UPDATE;
+BUSINESS_ATTRIBUTE_UPDATE, CUSTOM_ATTRIBUTE_UPDATE;
public static EntityAuditActionV2 fromString(String strValue) {
switch (strValue) {
@@ -97,6 +97,8 @@ public class EntityAuditEventV2 implements Serializable,
Clearable {
return LABEL_DELETE;
case "BUSINESS_ATTRIBUTE_UPDATE":
return BUSINESS_ATTRIBUTE_UPDATE;
+case "CUSTOM_ATTRIBUTE_UPDATE":
+return CUSTOM_ATTRIBUTE_UPDATE;
}
throw new IllegalArgumentException("No enum constant " +
EntityAuditActionV2.class.getCanonicalName() + "." + strValue);
diff --git
a/repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
b/repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
index ca6f373..0043f1c 100644
---
a/repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
+++
b/repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
@@ -63,6 +63,7 @@ import static
org.apache.atlas.model.audit.EntityAuditEventV2.EntityAuditActionV
import static
org.apache.atlas.model.audit.EntityAudi
[atlas] 01/02: ATLAS-4076: Propagated-tag-addition entries were appearing multiple times on derived table audit
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 9f20077ec9fc6ea672803b11f037fea1bad168e9
Author: Deep Singh
AuthorDate: Fri Dec 11 12:45:37 2020 -0600
ATLAS-4076: Propagated-tag-addition entries were appearing multiple times
on derived table audit
Signed-off-by: Ashutosh Mestry
---
.../repository/store/graph/v2/AtlasEntityChangeNotifier.java | 2 ++
server-api/src/main/java/org/apache/atlas/RequestContext.java | 8
2 files changed, 10 insertions(+)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
index 32ad65e..99be169 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
@@ -359,7 +359,9 @@ public class AtlasEntityChangeNotifier implements
IAtlasEntityChangeNotifier {
Map> removedPropagations =
context.getRemovedPropagations();
notifyPropagatedEntities(addedPropagations,
PROPAGATED_CLASSIFICATION_ADD);
+context.clearAddedPropagations();
notifyPropagatedEntities(removedPropagations,
PROPAGATED_CLASSIFICATION_DELETE);
+context.clearRemovePropagations();
}
@Override
diff --git a/server-api/src/main/java/org/apache/atlas/RequestContext.java
b/server-api/src/main/java/org/apache/atlas/RequestContext.java
index 32ffddd..7f0cfe5 100644
--- a/server-api/src/main/java/org/apache/atlas/RequestContext.java
+++ b/server-api/src/main/java/org/apache/atlas/RequestContext.java
@@ -329,6 +329,14 @@ public class RequestContext {
return deletedEntities.values();
}
+public void clearRemovePropagations() {
+removedPropagations.clear();
+}
+
+public void clearAddedPropagations() {
+addedPropagations.clear();
+}
+
/**
* Checks if an instance with the given guid is in the cache for this
request. Either returns the instance
* or null if it is not in the cache.
[atlas] 02/02: ATLAS-4067: Updates in Custom Attributes and Business Attributes must be captured separately in entity audits.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit c1c28c12add6bf4b19a8218037b52409970d9857
Author: Deep Singh
AuthorDate: Mon Dec 14 12:40:08 2020 -0600
ATLAS-4067: Updates in Custom Attributes and Business Attributes must be
captured separately in entity audits.
Signed-off-by: Ashutosh Mestry
---
dashboardv2/public/js/utils/Enums.js | 1 +
dashboardv3/public/js/utils/Enums.js | 1 +
.../apache/atlas/exception/AtlasBaseException.java | 1 -
.../atlas/model/audit/EntityAuditEventV2.java | 4 +++-
.../repository/audit/EntityAuditListenerV2.java| 13 +++-
.../store/graph/v2/AtlasEntityStoreV2.java | 18 +++-
.../main/java/org/apache/atlas/RequestContext.java | 24 ++
7 files changed, 54 insertions(+), 8 deletions(-)
diff --git a/dashboardv2/public/js/utils/Enums.js
b/dashboardv2/public/js/utils/Enums.js
index ebfceba..d2e476d 100644
--- a/dashboardv2/public/js/utils/Enums.js
+++ b/dashboardv2/public/js/utils/Enums.js
@@ -42,6 +42,7 @@ define(["require", "backbone"], function(require) {
BUSINESS_ATTRIBUTE_ADD: "Business Attribute(s) Added",
BUSINESS_ATTRIBUTE_UPDATE: "Business Attribute(s) Updated",
BUSINESS_ATTRIBUTE_DELETE: "Business Attribute(s) Deleted",
+CUSTOM_ATTRIBUTE_UPDATE: "User-defined Attribute(s) Updated",
TYPE_DEF_UPDATE: "Type Updated",
TYPE_DEF_CREATE: "Type Created",
TYPE_DEF_DELETE: "Type Deleted",
diff --git a/dashboardv3/public/js/utils/Enums.js
b/dashboardv3/public/js/utils/Enums.js
index ebfceba..d2e476d 100644
--- a/dashboardv3/public/js/utils/Enums.js
+++ b/dashboardv3/public/js/utils/Enums.js
@@ -42,6 +42,7 @@ define(["require", "backbone"], function(require) {
BUSINESS_ATTRIBUTE_ADD: "Business Attribute(s) Added",
BUSINESS_ATTRIBUTE_UPDATE: "Business Attribute(s) Updated",
BUSINESS_ATTRIBUTE_DELETE: "Business Attribute(s) Deleted",
+CUSTOM_ATTRIBUTE_UPDATE: "User-defined Attribute(s) Updated",
TYPE_DEF_UPDATE: "Type Updated",
TYPE_DEF_CREATE: "Type Created",
TYPE_DEF_DELETE: "Type Deleted",
diff --git
a/intg/src/main/java/org/apache/atlas/exception/AtlasBaseException.java
b/intg/src/main/java/org/apache/atlas/exception/AtlasBaseException.java
index 5538702..88a26f4 100644
--- a/intg/src/main/java/org/apache/atlas/exception/AtlasBaseException.java
+++ b/intg/src/main/java/org/apache/atlas/exception/AtlasBaseException.java
@@ -19,7 +19,6 @@ package org.apache.atlas.exception;
import org.apache.atlas.AtlasErrorCode;
-import javax.ws.rs.core.Response;
import java.util.List;
/**
diff --git
a/intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
b/intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
index c37e282..083acac 100644
--- a/intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
+++ b/intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
@@ -54,7 +54,7 @@ public class EntityAuditEventV2 implements Serializable,
Clearable {
CLASSIFICATION_ADD, CLASSIFICATION_DELETE, CLASSIFICATION_UPDATE,
PROPAGATED_CLASSIFICATION_ADD, PROPAGATED_CLASSIFICATION_DELETE,
PROPAGATED_CLASSIFICATION_UPDATE,
TERM_ADD, TERM_DELETE, LABEL_ADD, LABEL_DELETE, ENTITY_PURGE,
-BUSINESS_ATTRIBUTE_UPDATE;
+BUSINESS_ATTRIBUTE_UPDATE, CUSTOM_ATTRIBUTE_UPDATE;
public static EntityAuditActionV2 fromString(String strValue) {
switch (strValue) {
@@ -97,6 +97,8 @@ public class EntityAuditEventV2 implements Serializable,
Clearable {
return LABEL_DELETE;
case "BUSINESS_ATTRIBUTE_UPDATE":
return BUSINESS_ATTRIBUTE_UPDATE;
+case "CUSTOM_ATTRIBUTE_UPDATE":
+return CUSTOM_ATTRIBUTE_UPDATE;
}
throw new IllegalArgumentException("No enum constant " +
EntityAuditActionV2.class.getCanonicalName() + "." + strValue);
diff --git
a/repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
b/repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
index ca6f373..0043f1c 100644
---
a/repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
+++
b/repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
@@ -63,6 +63,7 @@ import static
org.apache.atlas.model.audit.EntityAuditEventV2.EntityAuditActionV
import static
org.apache.atlas.model.audit.EntityAudi
[atlas] 01/02: ATLAS-4076: Propagated-tag-addition entries were appearing multiple times on derived table audit
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 2e40de917083c7c5583a61537b6447ab3af41918
Author: Deep Singh
AuthorDate: Fri Dec 11 12:45:37 2020 -0600
ATLAS-4076: Propagated-tag-addition entries were appearing multiple times
on derived table audit
Signed-off-by: Ashutosh Mestry
---
.../repository/store/graph/v2/AtlasEntityChangeNotifier.java | 2 ++
server-api/src/main/java/org/apache/atlas/RequestContext.java | 8
2 files changed, 10 insertions(+)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
index 32ad65e..99be169 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
@@ -359,7 +359,9 @@ public class AtlasEntityChangeNotifier implements
IAtlasEntityChangeNotifier {
Map> removedPropagations =
context.getRemovedPropagations();
notifyPropagatedEntities(addedPropagations,
PROPAGATED_CLASSIFICATION_ADD);
+context.clearAddedPropagations();
notifyPropagatedEntities(removedPropagations,
PROPAGATED_CLASSIFICATION_DELETE);
+context.clearRemovePropagations();
}
@Override
diff --git a/server-api/src/main/java/org/apache/atlas/RequestContext.java
b/server-api/src/main/java/org/apache/atlas/RequestContext.java
index 32ffddd..7f0cfe5 100644
--- a/server-api/src/main/java/org/apache/atlas/RequestContext.java
+++ b/server-api/src/main/java/org/apache/atlas/RequestContext.java
@@ -329,6 +329,14 @@ public class RequestContext {
return deletedEntities.values();
}
+public void clearRemovePropagations() {
+removedPropagations.clear();
+}
+
+public void clearAddedPropagations() {
+addedPropagations.clear();
+}
+
/**
* Checks if an instance with the given guid is in the cache for this
request. Either returns the instance
* or null if it is not in the cache.
[atlas] branch master updated (de87800 -> c1c28c1)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git. from de87800 ATLAS-4058: UI: Fix Entity detail page Hive Column Position with 0 is set to NA. new 2e40de9 ATLAS-4076: Propagated-tag-addition entries were appearing multiple times on derived table audit new c1c28c1 ATLAS-4067: Updates in Custom Attributes and Business Attributes must be captured separately in entity audits. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: dashboardv2/public/js/utils/Enums.js | 1 + dashboardv3/public/js/utils/Enums.js | 1 + .../apache/atlas/exception/AtlasBaseException.java | 1 - .../atlas/model/audit/EntityAuditEventV2.java | 4 ++- .../repository/audit/EntityAuditListenerV2.java| 13 - .../store/graph/v2/AtlasEntityChangeNotifier.java | 2 ++ .../store/graph/v2/AtlasEntityStoreV2.java | 18 .../main/java/org/apache/atlas/RequestContext.java | 32 ++ 8 files changed, 64 insertions(+), 8 deletions(-)
[atlas] 02/02: ATLAS-4024: Export Service: Export of terms.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 5d3649c1b11e495547db22c192c163ff2a0f7044
Author: Ashutosh Mestry
AuthorDate: Wed Dec 9 21:52:59 2020 -0800
ATLAS-4024: Export Service: Export of terms.
Signed-off-by: Ashutosh Mestry
---
.../atlas/repository/impexp/ExportService.java | 43 --
.../repository/impexp/ExportTypeProcessor.java | 40 +++-
2 files changed, 79 insertions(+), 4 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
b/repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
index 8af2057..65d7a18 100644
---
a/repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
+++
b/repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
@@ -20,6 +20,7 @@ package org.apache.atlas.repository.impexp;
import com.google.common.annotations.VisibleForTesting;
import org.apache.atlas.RequestContext;
import org.apache.atlas.exception.AtlasBaseException;
+import org.apache.atlas.glossary.GlossaryService;
import org.apache.atlas.model.impexp.AtlasExportRequest;
import org.apache.atlas.model.impexp.AtlasExportResult;
import org.apache.atlas.model.instance.AtlasEntity;
@@ -38,6 +39,7 @@ import org.apache.atlas.repository.util.UniqueList;
import org.apache.atlas.type.AtlasTypeRegistry;
import org.apache.atlas.util.AtlasGremlinQueryProvider;
import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.collections.MapUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
@@ -65,14 +67,17 @@ public class ExportService {
private final EntityGraphRetriever entityGraphRetriever;
private ExportTypeProcessor exportTypeProcessor;
private final HdfsPathEntityCreator hdfsPathEntityCreator;
+private final GlossaryService glossaryService;
@Inject
public ExportService(final AtlasTypeRegistry typeRegistry, AtlasGraph
graph,
- AuditsWriter auditsWriter, HdfsPathEntityCreator
hdfsPathEntityCreator) {
+ AuditsWriter auditsWriter, HdfsPathEntityCreator
hdfsPathEntityCreator,
+ GlossaryService glossaryService) {
this.typeRegistry = typeRegistry;
this.entityGraphRetriever = new EntityGraphRetriever(graph,
this.typeRegistry);
this.auditsWriter = auditsWriter;
this.hdfsPathEntityCreator = hdfsPathEntityCreator;
+this.glossaryService = glossaryService;
this.startEntityFetchByExportRequest = new
StartEntityFetchByExportRequest(graph, typeRegistry,
AtlasGremlinQueryProvider.INSTANCE);
this.entitiesExtractor = new EntitiesExtractor(graph, typeRegistry);
}
@@ -84,7 +89,7 @@ public class ExportService {
hostName, startTime, getCurrentChangeMarker());
ExportContext context = new ExportContext(result, exportSink);
-exportTypeProcessor = new ExportTypeProcessor(typeRegistry);
+exportTypeProcessor = new ExportTypeProcessor(typeRegistry,
glossaryService);
try {
LOG.info("==> export(user={}, from={})", userName, requestingIP);
@@ -264,8 +269,12 @@ public class ExportService {
}
public void processEntity(AtlasEntityWithExtInfo entityWithExtInfo,
ExportContext context) throws AtlasBaseException {
-addEntity(entityWithExtInfo, context);
exportTypeProcessor.addTypes(entityWithExtInfo.getEntity(), context);
+if (MapUtils.isNotEmpty(context.termsGlossary)) {
+addGlossaryEntities(context);
+}
+
+addEntity(entityWithExtInfo, context);
context.guidsProcessed.add(entityWithExtInfo.getEntity().getGuid());
entitiesExtractor.get(entityWithExtInfo.getEntity(), context);
@@ -280,6 +289,28 @@ public class ExportService {
}
}
+private void addGlossaryEntities(ExportContext context) {
+try {
+for (String termGuid : context.termsGlossary.keySet()) {
+try {
+String glossaryGuid = context.termsGlossary.get(termGuid);
+if (!context.sink.hasEntity(glossaryGuid)) {
+AtlasEntity glossary =
entityGraphRetriever.toAtlasEntity(glossaryGuid);
+addEntity(new AtlasEntityWithExtInfo(glossary),
context);
+}
+
+if (!context.sink.hasEntity(termGuid)) {
+AtlasEntity term =
entityGraphRetriever.toAtlasEntity(termGuid);
+addEntity(new AtlasEntityWithExtInfo(term), context);
+}
+} catch (AtlasBaseExcept
[atlas] branch branch-2.0 updated (d9f8c88 -> 5d3649c)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git. from d9f8c88 Atlas-4062: UI:- Classification Daterange Picker Datetime format fix. new 97220fd ATLAS-4068: Export/Import: Conditionally Support Simpultaneous Operations. new 5d3649c ATLAS-4024: Export Service: Export of terms. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../atlas/repository/impexp/ExportService.java | 43 +++- .../repository/impexp/ExportTypeProcessor.java | 40 ++- .../apache/atlas/web/resources/AdminResource.java | 79 ++ 3 files changed, 131 insertions(+), 31 deletions(-)
[atlas] 02/02: ATLAS-4024: Export Service: Export of terms.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 31125c58c8a6d2edf0bd7de4e31c44e18adc9a7f
Author: Ashutosh Mestry
AuthorDate: Wed Dec 9 21:52:59 2020 -0800
ATLAS-4024: Export Service: Export of terms.
Signed-off-by: Ashutosh Mestry
---
.../atlas/repository/impexp/ExportService.java | 43 --
.../repository/impexp/ExportTypeProcessor.java | 40 +++-
2 files changed, 79 insertions(+), 4 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
b/repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
index 8af2057..65d7a18 100644
---
a/repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
+++
b/repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
@@ -20,6 +20,7 @@ package org.apache.atlas.repository.impexp;
import com.google.common.annotations.VisibleForTesting;
import org.apache.atlas.RequestContext;
import org.apache.atlas.exception.AtlasBaseException;
+import org.apache.atlas.glossary.GlossaryService;
import org.apache.atlas.model.impexp.AtlasExportRequest;
import org.apache.atlas.model.impexp.AtlasExportResult;
import org.apache.atlas.model.instance.AtlasEntity;
@@ -38,6 +39,7 @@ import org.apache.atlas.repository.util.UniqueList;
import org.apache.atlas.type.AtlasTypeRegistry;
import org.apache.atlas.util.AtlasGremlinQueryProvider;
import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.collections.MapUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
@@ -65,14 +67,17 @@ public class ExportService {
private final EntityGraphRetriever entityGraphRetriever;
private ExportTypeProcessor exportTypeProcessor;
private final HdfsPathEntityCreator hdfsPathEntityCreator;
+private final GlossaryService glossaryService;
@Inject
public ExportService(final AtlasTypeRegistry typeRegistry, AtlasGraph
graph,
- AuditsWriter auditsWriter, HdfsPathEntityCreator
hdfsPathEntityCreator) {
+ AuditsWriter auditsWriter, HdfsPathEntityCreator
hdfsPathEntityCreator,
+ GlossaryService glossaryService) {
this.typeRegistry = typeRegistry;
this.entityGraphRetriever = new EntityGraphRetriever(graph,
this.typeRegistry);
this.auditsWriter = auditsWriter;
this.hdfsPathEntityCreator = hdfsPathEntityCreator;
+this.glossaryService = glossaryService;
this.startEntityFetchByExportRequest = new
StartEntityFetchByExportRequest(graph, typeRegistry,
AtlasGremlinQueryProvider.INSTANCE);
this.entitiesExtractor = new EntitiesExtractor(graph, typeRegistry);
}
@@ -84,7 +89,7 @@ public class ExportService {
hostName, startTime, getCurrentChangeMarker());
ExportContext context = new ExportContext(result, exportSink);
-exportTypeProcessor = new ExportTypeProcessor(typeRegistry);
+exportTypeProcessor = new ExportTypeProcessor(typeRegistry,
glossaryService);
try {
LOG.info("==> export(user={}, from={})", userName, requestingIP);
@@ -264,8 +269,12 @@ public class ExportService {
}
public void processEntity(AtlasEntityWithExtInfo entityWithExtInfo,
ExportContext context) throws AtlasBaseException {
-addEntity(entityWithExtInfo, context);
exportTypeProcessor.addTypes(entityWithExtInfo.getEntity(), context);
+if (MapUtils.isNotEmpty(context.termsGlossary)) {
+addGlossaryEntities(context);
+}
+
+addEntity(entityWithExtInfo, context);
context.guidsProcessed.add(entityWithExtInfo.getEntity().getGuid());
entitiesExtractor.get(entityWithExtInfo.getEntity(), context);
@@ -280,6 +289,28 @@ public class ExportService {
}
}
+private void addGlossaryEntities(ExportContext context) {
+try {
+for (String termGuid : context.termsGlossary.keySet()) {
+try {
+String glossaryGuid = context.termsGlossary.get(termGuid);
+if (!context.sink.hasEntity(glossaryGuid)) {
+AtlasEntity glossary =
entityGraphRetriever.toAtlasEntity(glossaryGuid);
+addEntity(new AtlasEntityWithExtInfo(glossary),
context);
+}
+
+if (!context.sink.hasEntity(termGuid)) {
+AtlasEntity term =
entityGraphRetriever.toAtlasEntity(termGuid);
+addEntity(new AtlasEntityWithExtInfo(term), context);
+}
+} catch (AtlasBaseExcept
[atlas] 01/02: ATLAS-4068: Export/Import: Conditionally Support Simpultaneous Operations.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 9cdc31d5851db61296c7ff07ecb9bc85c26939fa
Author: Ashutosh Mestry
AuthorDate: Wed Dec 9 14:43:54 2020 -0800
ATLAS-4068: Export/Import: Conditionally Support Simpultaneous Operations.
Signed-off-by: Ashutosh Mestry
---
.../apache/atlas/web/resources/AdminResource.java | 79 ++
1 file changed, 52 insertions(+), 27 deletions(-)
diff --git
a/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
b/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
index f8c953b..b20b404 100755
--- a/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
+++ b/webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
@@ -22,7 +22,6 @@ import com.sun.jersey.multipart.FormDataParam;
import org.apache.atlas.ApplicationProperties;
import org.apache.atlas.AtlasClient;
import org.apache.atlas.AtlasErrorCode;
-import org.apache.atlas.RequestContext;
import org.apache.atlas.authorize.AtlasAdminAccessRequest;
import org.apache.atlas.authorize.AtlasAuthorizationUtils;
import org.apache.atlas.authorize.AtlasEntityAccessRequest;
@@ -102,7 +101,6 @@ import java.io.InputStream;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
-import java.util.Date;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
@@ -392,7 +390,12 @@ public class AdminResource {
AtlasAuthorizationUtils.verifyAccess(new
AtlasAdminAccessRequest(AtlasPrivilege.ADMIN_EXPORT), "export");
-acquireExportImportLock("export");
+boolean preventMultipleRequests = request != null
+&&
!(request.getOptions().containsKey(AtlasExportRequest.OPTION_SKIP_LINEAGE)
+ ||
request.getOptions().containsKey(AtlasExportRequest.OPTION_KEY_REPLICATED_TO));
+if (preventMultipleRequests) {
+acquireExportImportLock("export");
+}
ZipSink exportSink = null;
boolean isSuccessful = false;
@@ -419,22 +422,15 @@ public class AdminResource {
throw new AtlasBaseException(excp);
} finally {
-releaseExportImportLock();
+if (preventMultipleRequests) {
+releaseExportImportLock();
+}
if (exportSink != null) {
exportSink.close();
}
-if (isSuccessful &&
CollectionUtils.isNotEmpty(result.getRequest().getItemsToExport())) {
-
-Map optionMap =
result.getRequest().getOptions();
-optionMap.put(OPERATION_STATUS,
result.getOperationStatus().name());
-String params = AtlasJson.toJson(optionMap);
-
-List objectIds =
result.getRequest().getItemsToExport();
-
-auditImportExportOperations(objectIds, AuditOperation.EXPORT,
params);
-}
+addToExportOperationAudits(isSuccessful, result);
if (LOG.isDebugEnabled()) {
LOG.debug("<== AdminResource.export()");
@@ -454,11 +450,15 @@ public class AdminResource {
AtlasAuthorizationUtils.verifyAccess(new
AtlasAdminAccessRequest(AtlasPrivilege.ADMIN_IMPORT), "importData");
-acquireExportImportLock("import");
AtlasImportResult result = null;
+boolean preventMultipleRequests = true;
try {
AtlasImportRequest request = AtlasType.fromJson(jsonData,
AtlasImportRequest.class);
+preventMultipleRequests = request != null &&
!request.getOptions().containsKey(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM);
+if (preventMultipleRequests) {
+acquireExportImportLock("import");
+}
result = importService.run(inputStream, request,
Servlets.getUserName(httpServletRequest),
Servlets.getHostName(httpServletRequest),
@@ -477,20 +477,16 @@ public class AdminResource {
throw new AtlasBaseException(excp);
} finally {
-releaseExportImportLock();
+if (preventMultipleRequests) {
+releaseExportImportLock();
+}
if (LOG.isDebugEnabled()) {
LOG.debug("<== AdminResource.importData(binary)");
}
}
-List objectIds =
result.getExportResult().getRequest().getItemsToExport();
-
-Map optionMap = new HashMap<>();
-optionMap.put(OPERATION_STATUS, result.getOperationStatus().name());
-String params = AtlasJson.toJson(optionMap);
-
-auditImportExportOperations(objectIds, AuditOperation.IMPORT, params);
+a
[atlas] branch master updated (ff55b0a -> 31125c5)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git. from ff55b0a Atlas-4062: UI:- Classification Daterange Picker Datetime format fix. new 9cdc31d ATLAS-4068: Export/Import: Conditionally Support Simpultaneous Operations. new 31125c5 ATLAS-4024: Export Service: Export of terms. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../atlas/repository/impexp/ExportService.java | 43 +++- .../repository/impexp/ExportTypeProcessor.java | 40 ++- .../apache/atlas/web/resources/AdminResource.java | 79 ++ 3 files changed, 131 insertions(+), 31 deletions(-)
[atlas] 02/02: ATLAS-4049: Few admin apis were not redirected to Active server in HA mode
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit e464779787eebfec65b486d26508659f593a4abd
Author: Deep Singh
AuthorDate: Thu Dec 3 12:01:37 2020 -0600
ATLAS-4049: Few admin apis were not redirected to Active server in HA mode
Signed-off-by: Ashutosh Mestry
---
.../main/java/org/apache/atlas/web/filters/ActiveServerFilter.java| 3 ++-
.../java/org/apache/atlas/web/filters/ActiveServerFilterTest.java | 4 +++-
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git
a/webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
b/webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
index 2c28aaf..98eb3d3 100644
--- a/webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
+++ b/webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
@@ -109,7 +109,8 @@ public class ActiveServerFilter implements Filter {
}
}
-final String adminUriNotFiltered[] = { "/admin/export", "/admin/import",
"/admin/importfile" };
+final String adminUriNotFiltered[] = { "/admin/export", "/admin/import",
"/admin/importfile", "/admin/audits",
+"/admin/purge", "/admin/expimp/audit", "/admin/metrics",
"/admin/server", "/admin/audit/" };
private boolean isFilteredURI(ServletRequest servletRequest) {
HttpServletRequest httpServletRequest = (HttpServletRequest)
servletRequest;
String requestURI = httpServletRequest.getRequestURI();
diff --git
a/webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java
b/webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java
index d7ae0f2..defa5bb 100644
---
a/webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java
+++
b/webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java
@@ -105,7 +105,9 @@ public class ActiveServerFilterTest {
@Test
public void
adminImportRequestsToPassiveServerShouldToActiveServerAddress() throws
IOException, ServletException {
-String importExportUrls[] = {"api/admin/export", "api/admin/import",
"api/admin/importfile"};
+String importExportUrls[] = {"api/admin/export", "api/admin/import",
"api/admin/importfile", "api/admin/audits",
+"api/admin/purge", "api/admin/expimp/audit",
"api/admin/metrics",
+"api/admin/server/dummy_name",
"api/admin/audit/dummy_guid/details"};
for (String partialUrl : importExportUrls) {
when(serviceState.getState()).thenReturn(ServiceState.ServiceStateValue.PASSIVE);
[atlas] branch branch-2.0 updated (09e94cc -> e464779)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git. from 09e94cc ATLAS-3913 Swagger documentation for APIs new 41e83f1 ATLAS-4059 rename the config flag for enabling internal kafka topic import new e464779 ATLAS-4049: Few admin apis were not redirected to Active server in HA mode The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: common/src/main/java/org/apache/atlas/utils/KafkaUtils.java | 2 +- .../main/java/org/apache/atlas/web/filters/ActiveServerFilter.java| 3 ++- .../java/org/apache/atlas/web/filters/ActiveServerFilterTest.java | 4 +++- 3 files changed, 6 insertions(+), 3 deletions(-)
[atlas] 01/02: ATLAS-4059 rename the config flag for enabling internal kafka topic import
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 41e83f13526540e7864b1c2b55b74dc9aca503e0
Author: Deep Singh
AuthorDate: Thu Dec 3 12:20:31 2020 -0600
ATLAS-4059 rename the config flag for enabling internal kafka topic import
Signed-off-by: Ashutosh Mestry
---
common/src/main/java/org/apache/atlas/utils/KafkaUtils.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
b/common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
index 7e9f789..14e205a 100644
--- a/common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
+++ b/common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
@@ -58,7 +58,7 @@ public class KafkaUtils implements AutoCloseable {
private static final String JAAS_PRINCIPAL_PROP = "principal";
private static final String JAAS_DEFAULT_CLIENT_NAME = "KafkaClient";
private static final String JAAS_TICKET_BASED_CLIENT_NAME =
"ticketBased-KafkaClient";
-private static final String IMPORT_INTERNAL_TOPICS =
"atlas.hook.kafka.import.internal.topics";
+private static final String IMPORT_INTERNAL_TOPICS =
"atlas.kafka.bridge.enable.internal.topics.import";
public static final String ATLAS_KAFKA_PROPERTY_PREFIX = "atlas.kafka";
[atlas] 01/02: ATLAS-4059 rename the config flag for enabling internal kafka topic import
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 8801ba6814abbd2c60c88d04cf1649efed2f3429
Author: Deep Singh
AuthorDate: Thu Dec 3 12:20:31 2020 -0600
ATLAS-4059 rename the config flag for enabling internal kafka topic import
Signed-off-by: Ashutosh Mestry
---
common/src/main/java/org/apache/atlas/utils/KafkaUtils.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
b/common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
index 7e9f789..14e205a 100644
--- a/common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
+++ b/common/src/main/java/org/apache/atlas/utils/KafkaUtils.java
@@ -58,7 +58,7 @@ public class KafkaUtils implements AutoCloseable {
private static final String JAAS_PRINCIPAL_PROP = "principal";
private static final String JAAS_DEFAULT_CLIENT_NAME = "KafkaClient";
private static final String JAAS_TICKET_BASED_CLIENT_NAME =
"ticketBased-KafkaClient";
-private static final String IMPORT_INTERNAL_TOPICS =
"atlas.hook.kafka.import.internal.topics";
+private static final String IMPORT_INTERNAL_TOPICS =
"atlas.kafka.bridge.enable.internal.topics.import";
public static final String ATLAS_KAFKA_PROPERTY_PREFIX = "atlas.kafka";
[atlas] 02/02: ATLAS-4049: Few admin apis were not redirected to Active server in HA mode
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 10b6d142e479e0b2a27be6a7dd8b9f82a4d3f723
Author: Deep Singh
AuthorDate: Thu Dec 3 12:01:37 2020 -0600
ATLAS-4049: Few admin apis were not redirected to Active server in HA mode
Signed-off-by: Ashutosh Mestry
---
.../main/java/org/apache/atlas/web/filters/ActiveServerFilter.java| 3 ++-
.../java/org/apache/atlas/web/filters/ActiveServerFilterTest.java | 4 +++-
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git
a/webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
b/webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
index 2c28aaf..98eb3d3 100644
--- a/webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
+++ b/webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
@@ -109,7 +109,8 @@ public class ActiveServerFilter implements Filter {
}
}
-final String adminUriNotFiltered[] = { "/admin/export", "/admin/import",
"/admin/importfile" };
+final String adminUriNotFiltered[] = { "/admin/export", "/admin/import",
"/admin/importfile", "/admin/audits",
+"/admin/purge", "/admin/expimp/audit", "/admin/metrics",
"/admin/server", "/admin/audit/" };
private boolean isFilteredURI(ServletRequest servletRequest) {
HttpServletRequest httpServletRequest = (HttpServletRequest)
servletRequest;
String requestURI = httpServletRequest.getRequestURI();
diff --git
a/webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java
b/webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java
index d7ae0f2..defa5bb 100644
---
a/webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java
+++
b/webapp/src/test/java/org/apache/atlas/web/filters/ActiveServerFilterTest.java
@@ -105,7 +105,9 @@ public class ActiveServerFilterTest {
@Test
public void
adminImportRequestsToPassiveServerShouldToActiveServerAddress() throws
IOException, ServletException {
-String importExportUrls[] = {"api/admin/export", "api/admin/import",
"api/admin/importfile"};
+String importExportUrls[] = {"api/admin/export", "api/admin/import",
"api/admin/importfile", "api/admin/audits",
+"api/admin/purge", "api/admin/expimp/audit",
"api/admin/metrics",
+"api/admin/server/dummy_name",
"api/admin/audit/dummy_guid/details"};
for (String partialUrl : importExportUrls) {
when(serviceState.getState()).thenReturn(ServiceState.ServiceStateValue.PASSIVE);
[atlas] branch master updated (294ca40 -> 10b6d14)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git. from 294ca40 ATLAS-3913 Swagger documentation for APIs new 8801ba6 ATLAS-4059 rename the config flag for enabling internal kafka topic import new 10b6d14 ATLAS-4049: Few admin apis were not redirected to Active server in HA mode The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: common/src/main/java/org/apache/atlas/utils/KafkaUtils.java | 2 +- .../main/java/org/apache/atlas/web/filters/ActiveServerFilter.java| 3 ++- .../java/org/apache/atlas/web/filters/ActiveServerFilterTest.java | 4 +++- 3 files changed, 6 insertions(+), 3 deletions(-)
[atlas] branch branch-2.0 updated: ATLAS-4048: Import Service: Entity Update: Businessmetada changes.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 5ee9421 ATLAS-4048: Import Service: Entity Update: Businessmetada
changes.
5ee9421 is described below
commit 5ee942188d23f23d7fab83595b8abdc16af53ff1
Author: Ashutosh Mestry
AuthorDate: Tue Nov 24 21:04:22 2020 -0800
ATLAS-4048: Import Service: Entity Update: Businessmetada changes.
---
.../atlas/repository/store/graph/v2/AtlasEntityStoreV2.java | 9 +
1 file changed, 9 insertions(+)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
index 7cf77ea..b80e42e 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
@@ -1208,6 +1208,15 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
}
}
+if (!hasUpdates && replaceBusinessAttributes) {
+Map> currBusinessMetadata
= entityRetriever.getBusinessMetadata(vertex);
+Map> newBusinessMetadata
= entity.getBusinessAttributes();
+
+if (!Objects.equals(currBusinessMetadata,
newBusinessMetadata)) {
+hasUpdates = true;
+}
+}
+
// if classifications are to be replaced, then skip
updates only when no change in classifications
if (!hasUpdates && replaceClassifications) {
List newVal =
entity.getClassifications();
[atlas] branch master updated: ATLAS-4048: Import Service: Entity Update: Businessmetada changes.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 5373cde ATLAS-4048: Import Service: Entity Update: Businessmetada
changes.
5373cde is described below
commit 5373cde5f4624e0989a41e9512e67f903d1fc26c
Author: Ashutosh Mestry
AuthorDate: Tue Nov 24 21:04:22 2020 -0800
ATLAS-4048: Import Service: Entity Update: Businessmetada changes.
---
.../atlas/repository/store/graph/v2/AtlasEntityStoreV2.java | 9 +
1 file changed, 9 insertions(+)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
index 7cf77ea..b80e42e 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
@@ -1208,6 +1208,15 @@ public class AtlasEntityStoreV2 implements
AtlasEntityStore {
}
}
+if (!hasUpdates && replaceBusinessAttributes) {
+Map> currBusinessMetadata
= entityRetriever.getBusinessMetadata(vertex);
+Map> newBusinessMetadata
= entity.getBusinessAttributes();
+
+if (!Objects.equals(currBusinessMetadata,
newBusinessMetadata)) {
+hasUpdates = true;
+}
+}
+
// if classifications are to be replaced, then skip
updates only when no change in classifications
if (!hasUpdates && replaceClassifications) {
List newVal =
entity.getClassifications();
[atlas] 02/02: ATLAS-3941: Notification Consumer: Reduce pause time on AtlasSchemaViolation exception.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 998e762c643b8628e0f46f8aff92777e9942fd52
Author: Ashutosh Mestry
AuthorDate: Wed Nov 18 13:53:38 2020 -0800
ATLAS-3941: Notification Consumer: Reduce pause time on
AtlasSchemaViolation exception.
---
.../org/apache/atlas/notification/NotificationHookConsumer.java| 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git
a/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
b/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
index f02c05f..0e58dac 100644
---
a/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
+++
b/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
@@ -809,9 +809,10 @@ public class NotificationHookConsumer implements Service,
ActiveStateChangeHandl
recordFailedMessages();
}
return;
-} else if (e instanceof
org.apache.atlas.repository.graphdb.AtlasSchemaViolationException
-||
exceptionClassName.equals(EXCEPTION_CLASS_NAME_JANUSGRAPH_EXCEPTION)
-||
exceptionClassName.equals(EXCEPTION_CLASS_NAME_PERMANENTLOCKING_EXCEPTION)) {
+} else if (e instanceof
org.apache.atlas.repository.graphdb.AtlasSchemaViolationException) {
+LOG.warn("{}: Continuing: {}", exceptionClassName,
e.getMessage());
+} else if
(exceptionClassName.equals(EXCEPTION_CLASS_NAME_JANUSGRAPH_EXCEPTION)
+||
exceptionClassName.equals(EXCEPTION_CLASS_NAME_PERMANENTLOCKING_EXCEPTION)) {
LOG.warn("{}: Pausing & retry: Try: {}: Pause: {}
ms. {}",
exceptionClassName, numRetries,
adaptiveWaiter.waitDuration, e.getMessage());
[atlas] branch branch-2.0 updated (9734248 -> 998e762)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/atlas.git. from 9734248 ATLAS-4045: fixed typos and other grammatical mistakes in export/import API documentation new a0f2af8 ATLAS-4025 ATLAS-4023: Import Service: Lables and Classifications not getting updated. new 998e762 ATLAS-3941: Notification Consumer: Reduce pause time on AtlasSchemaViolation exception. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/atlas/repository/store/graph/v2/EntityGraphMapper.java | 7 +-- .../org/apache/atlas/notification/NotificationHookConsumer.java| 7 --- 2 files changed, 9 insertions(+), 5 deletions(-)
[atlas] 01/02: ATLAS-4025 ATLAS-4023: Import Service: Lables and Classifications not getting updated.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit a0f2af87e271536c97820b26ede8dc7c845323fa
Author: Ashutosh Mestry
AuthorDate: Fri Nov 13 10:26:25 2020 -0800
ATLAS-4025 ATLAS-4023: Import Service: Lables and Classifications not
getting updated.
---
.../apache/atlas/repository/store/graph/v2/EntityGraphMapper.java | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
index 14876b1..2cfcc0b 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
@@ -255,6 +255,10 @@ public class EntityGraphMapper {
if (entity.getCustomAttributes() != null) {
setCustomAttributes(vertex, entity);
}
+
+if (entity.getLabels() != null) {
+setLabels(vertex, entity.getLabels());
+}
}
public EntityMutationResponse
mapAttributesAndClassifications(EntityMutationContext context, final boolean
isPartialUpdate, final boolean replaceClassifications, boolean
replaceBusinessAttributes) throws AtlasBaseException {
@@ -299,8 +303,6 @@ public class EntityGraphMapper {
mapAttributes(updatedEntity, entityType, vertex, updateType,
context);
setCustomAttributes(vertex,updatedEntity);
-resp.addEntity(updateType, constructHeader(updatedEntity,
vertex));
-
if (replaceClassifications) {
deleteClassifications(guid);
addClassifications(context, guid,
updatedEntity.getClassifications());
@@ -310,6 +312,7 @@ public class EntityGraphMapper {
setBusinessAttributes(vertex, entityType,
updatedEntity.getBusinessAttributes());
}
+resp.addEntity(updateType, constructHeader(updatedEntity,
vertex));
reqContext.cache(updatedEntity);
}
}
[atlas] branch master updated: ATLAS-3941: Notification Consumer: Reduce pause time on AtlasSchemaViolation exception.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new f4b1c42 ATLAS-3941: Notification Consumer: Reduce pause time on
AtlasSchemaViolation exception.
f4b1c42 is described below
commit f4b1c4294f37a53d2041fb925dc43dba5f166b67
Author: Ashutosh Mestry
AuthorDate: Wed Nov 18 13:53:38 2020 -0800
ATLAS-3941: Notification Consumer: Reduce pause time on
AtlasSchemaViolation exception.
---
.../org/apache/atlas/notification/NotificationHookConsumer.java| 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git
a/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
b/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
index f02c05f..0e58dac 100644
---
a/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
+++
b/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
@@ -809,9 +809,10 @@ public class NotificationHookConsumer implements Service,
ActiveStateChangeHandl
recordFailedMessages();
}
return;
-} else if (e instanceof
org.apache.atlas.repository.graphdb.AtlasSchemaViolationException
-||
exceptionClassName.equals(EXCEPTION_CLASS_NAME_JANUSGRAPH_EXCEPTION)
-||
exceptionClassName.equals(EXCEPTION_CLASS_NAME_PERMANENTLOCKING_EXCEPTION)) {
+} else if (e instanceof
org.apache.atlas.repository.graphdb.AtlasSchemaViolationException) {
+LOG.warn("{}: Continuing: {}", exceptionClassName,
e.getMessage());
+} else if
(exceptionClassName.equals(EXCEPTION_CLASS_NAME_JANUSGRAPH_EXCEPTION)
+||
exceptionClassName.equals(EXCEPTION_CLASS_NAME_PERMANENTLOCKING_EXCEPTION)) {
LOG.warn("{}: Pausing & retry: Try: {}: Pause: {}
ms. {}",
exceptionClassName, numRetries,
adaptiveWaiter.waitDuration, e.getMessage());
[atlas] branch master updated (06106fe -> 4ab95b9)
This is an automated email from the ASF dual-hosted git repository. amestry pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/atlas.git. from 06106fe ATLAS-4010: Sort maven properties lexicographically in pom for better readability new a0a3c31 ATLAS-4015: Add Re-indexing as JAVA_PATCH. Part 2 new 4ab95b9 ATLAS-4025 ATLAS-4023: Import Service: Lables and Classifications not getting updated. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/main/java/org/apache/atlas/AtlasConfiguration.java | 2 +- .../org/apache/atlas/repository/patches/ReIndexPatch.java | 14 -- .../atlas/repository/store/graph/v2/EntityGraphMapper.java | 7 +-- 3 files changed, 14 insertions(+), 9 deletions(-)
[atlas] 01/02: ATLAS-4015: Add Re-indexing as JAVA_PATCH. Part 2
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit a0a3c3183e2cc794eb35392d43008d0937616ffc
Author: Ashutosh Mestry
AuthorDate: Fri Nov 13 10:15:23 2020 -0800
ATLAS-4015: Add Re-indexing as JAVA_PATCH. Part 2
---
.../src/main/java/org/apache/atlas/AtlasConfiguration.java | 2 +-
.../org/apache/atlas/repository/patches/ReIndexPatch.java | 14 --
2 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
index 2d3dfdf..ea9f26d 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
@@ -74,7 +74,7 @@ public enum AtlasConfiguration {
HTTP_HEADER_SERVER_VALUE("atlas.http.header.server.value","Apache Atlas"),
STORAGE_CONSISTENCY_LOCK_ENABLED("atlas.graph.storage.consistency-lock.enabled",
true),
-REINDEX_PATCH_ENABLED("atlas.patch.reindex.enabled", false);
+REBUILD_INDEX("atlas.rebuild.index", false);
private static final Configuration APPLICATION_PROPERTIES;
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/ReIndexPatch.java
b/repository/src/main/java/org/apache/atlas/repository/patches/ReIndexPatch.java
index a47a2cc..38d230f 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/ReIndexPatch.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/ReIndexPatch.java
@@ -23,6 +23,7 @@ import org.apache.atlas.pc.WorkItemBuilder;
import org.apache.atlas.pc.WorkItemConsumer;
import org.apache.atlas.pc.WorkItemManager;
import org.apache.atlas.repository.Constants;
+import org.apache.atlas.repository.graphdb.AtlasEdge;
import org.apache.atlas.repository.graphdb.AtlasElement;
import org.apache.atlas.repository.graphdb.AtlasGraph;
import org.apache.atlas.repository.graphdb.AtlasVertex;
@@ -53,7 +54,8 @@ public class ReIndexPatch extends AtlasPatchHandler {
@Override
public void apply() throws AtlasBaseException {
-if (AtlasConfiguration.REINDEX_PATCH_ENABLED.getBoolean() == false) {
+if (AtlasConfiguration.REBUILD_INDEX.getBoolean() == false) {
+LOG.info("ReIndexPatch: Skipped, since not enabled!");
return;
}
@@ -113,8 +115,8 @@ public class ReIndexPatch extends AtlasPatchHandler {
}
private static void edges(WorkItemManager manager, AtlasGraph graph) {
-Iterable iterable = graph.getEdges();
-for (Iterator iter = iterable.iterator(); iter.hasNext();
) {
+Iterable iterable = graph.getEdges();
+for (Iterator iter = iterable.iterator();
iter.hasNext(); ) {
manager.checkProduce(iter.next());
}
}
@@ -144,9 +146,9 @@ public class ReIndexPatch extends AtlasPatchHandler {
}
private static class ReindexConsumer extends
WorkItemConsumer {
-private List list = new ArrayList();
-private AtlasGraph graph;
-private String[] indexNames;
+private final List list = new ArrayList();
+private final String[] indexNames;
+private final AtlasGraph graph;
private final AtomicLong counter;
public ReindexConsumer(BlockingQueue queue, AtlasGraph graph, String[]
indexNames) {
[atlas] 02/02: ATLAS-4025 ATLAS-4023: Import Service: Lables and Classifications not getting updated.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
commit 4ab95b9adafd3d845b626d2d82bd2b3aa23be691
Author: Ashutosh Mestry
AuthorDate: Fri Nov 13 10:26:25 2020 -0800
ATLAS-4025 ATLAS-4023: Import Service: Lables and Classifications not
getting updated.
---
.../apache/atlas/repository/store/graph/v2/EntityGraphMapper.java | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
index 14876b1..2cfcc0b 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
@@ -255,6 +255,10 @@ public class EntityGraphMapper {
if (entity.getCustomAttributes() != null) {
setCustomAttributes(vertex, entity);
}
+
+if (entity.getLabels() != null) {
+setLabels(vertex, entity.getLabels());
+}
}
public EntityMutationResponse
mapAttributesAndClassifications(EntityMutationContext context, final boolean
isPartialUpdate, final boolean replaceClassifications, boolean
replaceBusinessAttributes) throws AtlasBaseException {
@@ -299,8 +303,6 @@ public class EntityGraphMapper {
mapAttributes(updatedEntity, entityType, vertex, updateType,
context);
setCustomAttributes(vertex,updatedEntity);
-resp.addEntity(updateType, constructHeader(updatedEntity,
vertex));
-
if (replaceClassifications) {
deleteClassifications(guid);
addClassifications(context, guid,
updatedEntity.getClassifications());
@@ -310,6 +312,7 @@ public class EntityGraphMapper {
setBusinessAttributes(vertex, entityType,
updatedEntity.getBusinessAttributes());
}
+resp.addEntity(updateType, constructHeader(updatedEntity,
vertex));
reqContext.cache(updatedEntity);
}
}
[atlas] branch branch-2.0 updated: ATLAS-4015: Add Re-indexing as JAVA_PATCH.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 9854ceb ATLAS-4015: Add Re-indexing as JAVA_PATCH.
9854ceb is described below
commit 9854ceb909401019c31b085d957b7200ba5e997d
Author: Ashutosh Mestry
AuthorDate: Tue Nov 10 09:47:00 2020 -0800
ATLAS-4015: Add Re-indexing as JAVA_PATCH.
---
.../repository/graphdb/AtlasGraphManagement.java | 8 +
.../graphdb/janus/AtlasJanusGraphManagement.java | 64 +++
.../java/org/apache/atlas/AtlasConfiguration.java | 3 +-
.../repository/patches/AtlasPatchManager.java | 3 +-
.../patches/ConcurrentPatchProcessor.java | 6 +-
.../atlas/repository/patches/ReIndexPatch.java | 199 +
6 files changed, 278 insertions(+), 5 deletions(-)
diff --git
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
index f7d2e27..7e3b2f4 100644
---
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
+++
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
@@ -172,4 +172,12 @@ public interface AtlasGraphManagement {
* Set consistency to ConsistencyModifier.LOCK for all vertex and edge
indexes.
*/
void updateUniqueIndexesForConsistencyLock();
+
+/***
+ * Re-index elements.
+ * @param indexName: Name of the index that needs to be operated on.
+ * @param elements: Elements to be re-indexed.
+ * @throws Exception
+ */
+void reindex(String indexName, List elements) throws
Exception;
}
diff --git
a/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
b/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
index 2a2ef92..1cc7f8b 100644
---
a/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
+++
b/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
@@ -19,14 +19,22 @@ package org.apache.atlas.repository.graphdb.janus;
import com.google.common.base.Preconditions;
import org.apache.atlas.AtlasConfiguration;
+import org.apache.atlas.exception.AtlasBaseException;
import org.apache.atlas.repository.graphdb.AtlasEdgeDirection;
+import org.apache.atlas.repository.graphdb.AtlasElement;
import org.apache.tinkerpop.gremlin.structure.Direction;
import org.apache.tinkerpop.gremlin.structure.Element;
import org.janusgraph.core.Cardinality;
import org.janusgraph.core.EdgeLabel;
+import org.janusgraph.core.JanusGraphElement;
import org.janusgraph.core.PropertyKey;
import org.janusgraph.core.schema.*;
import org.janusgraph.core.schema.JanusGraphManagement.IndexBuilder;
+import org.janusgraph.diskstorage.BackendTransaction;
+import org.janusgraph.diskstorage.indexing.IndexEntry;
+import org.janusgraph.graphdb.database.IndexSerializer;
+import org.janusgraph.graphdb.database.StandardJanusGraph;
+import org.janusgraph.graphdb.database.management.ManagementSystem;
import org.janusgraph.graphdb.internal.Token;
import org.apache.atlas.repository.graphdb.AtlasCardinality;
import org.apache.atlas.repository.graphdb.AtlasEdgeLabel;
@@ -36,11 +44,16 @@ import org.apache.atlas.repository.graphdb.AtlasPropertyKey;
import org.apache.commons.lang.StringUtils;
import org.apache.tinkerpop.gremlin.structure.Edge;
import org.apache.tinkerpop.gremlin.structure.Vertex;
+import org.janusgraph.graphdb.transaction.StandardJanusGraphTx;
+import org.janusgraph.graphdb.types.IndexType;
+import org.janusgraph.graphdb.types.MixedIndexType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
+import java.util.Map;
import java.util.Set;
/**
@@ -308,4 +321,55 @@ public class AtlasJanusGraphManagement implements
AtlasGraphManagement {
LOG.info("setConsistency: {}: {}: Done!",
elementType.getSimpleName(), count);
}
}
+
+@Override
+public void reindex(String indexName, List elements) throws
Exception {
+try {
+JanusGraphIndex index = management.getGraphIndex(indexName);
+if (index == null || !(management instanceof ManagementSystem) ||
!(graph.getGraph() instanceof StandardJanusGraph)) {
+LOG.error("Could not retrieve index for name: {} ", indexName);
+return;
+}
+
+ManagementSystem managementSystem = (ManagementSystem) management;
+IndexType indexType =
managementSystem.getSchemaVertex(index).asIndexType();
+if (!(indexType insta
[atlas] branch master updated: ATLAS-4015: Add Re-indexing as JAVA_PATCH.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new dc9e268 ATLAS-4015: Add Re-indexing as JAVA_PATCH.
dc9e268 is described below
commit dc9e268966fa27df400e827fd6e57a5098d7d478
Author: Ashutosh Mestry
AuthorDate: Tue Nov 10 09:47:00 2020 -0800
ATLAS-4015: Add Re-indexing as JAVA_PATCH.
---
.../repository/graphdb/AtlasGraphManagement.java | 8 +
.../graphdb/janus/AtlasJanusGraphManagement.java | 64 +++
.../java/org/apache/atlas/AtlasConfiguration.java | 3 +-
.../repository/patches/AtlasPatchManager.java | 3 +-
.../patches/ConcurrentPatchProcessor.java | 6 +-
.../atlas/repository/patches/ReIndexPatch.java | 199 +
6 files changed, 278 insertions(+), 5 deletions(-)
diff --git
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
index f7d2e27..7e3b2f4 100644
---
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
+++
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
@@ -172,4 +172,12 @@ public interface AtlasGraphManagement {
* Set consistency to ConsistencyModifier.LOCK for all vertex and edge
indexes.
*/
void updateUniqueIndexesForConsistencyLock();
+
+/***
+ * Re-index elements.
+ * @param indexName: Name of the index that needs to be operated on.
+ * @param elements: Elements to be re-indexed.
+ * @throws Exception
+ */
+void reindex(String indexName, List elements) throws
Exception;
}
diff --git
a/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
b/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
index 2a2ef92..1cc7f8b 100644
---
a/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
+++
b/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphManagement.java
@@ -19,14 +19,22 @@ package org.apache.atlas.repository.graphdb.janus;
import com.google.common.base.Preconditions;
import org.apache.atlas.AtlasConfiguration;
+import org.apache.atlas.exception.AtlasBaseException;
import org.apache.atlas.repository.graphdb.AtlasEdgeDirection;
+import org.apache.atlas.repository.graphdb.AtlasElement;
import org.apache.tinkerpop.gremlin.structure.Direction;
import org.apache.tinkerpop.gremlin.structure.Element;
import org.janusgraph.core.Cardinality;
import org.janusgraph.core.EdgeLabel;
+import org.janusgraph.core.JanusGraphElement;
import org.janusgraph.core.PropertyKey;
import org.janusgraph.core.schema.*;
import org.janusgraph.core.schema.JanusGraphManagement.IndexBuilder;
+import org.janusgraph.diskstorage.BackendTransaction;
+import org.janusgraph.diskstorage.indexing.IndexEntry;
+import org.janusgraph.graphdb.database.IndexSerializer;
+import org.janusgraph.graphdb.database.StandardJanusGraph;
+import org.janusgraph.graphdb.database.management.ManagementSystem;
import org.janusgraph.graphdb.internal.Token;
import org.apache.atlas.repository.graphdb.AtlasCardinality;
import org.apache.atlas.repository.graphdb.AtlasEdgeLabel;
@@ -36,11 +44,16 @@ import org.apache.atlas.repository.graphdb.AtlasPropertyKey;
import org.apache.commons.lang.StringUtils;
import org.apache.tinkerpop.gremlin.structure.Edge;
import org.apache.tinkerpop.gremlin.structure.Vertex;
+import org.janusgraph.graphdb.transaction.StandardJanusGraphTx;
+import org.janusgraph.graphdb.types.IndexType;
+import org.janusgraph.graphdb.types.MixedIndexType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
+import java.util.Map;
import java.util.Set;
/**
@@ -308,4 +321,55 @@ public class AtlasJanusGraphManagement implements
AtlasGraphManagement {
LOG.info("setConsistency: {}: {}: Done!",
elementType.getSimpleName(), count);
}
}
+
+@Override
+public void reindex(String indexName, List elements) throws
Exception {
+try {
+JanusGraphIndex index = management.getGraphIndex(indexName);
+if (index == null || !(management instanceof ManagementSystem) ||
!(graph.getGraph() instanceof StandardJanusGraph)) {
+LOG.error("Could not retrieve index for name: {} ", indexName);
+return;
+}
+
+ManagementSystem managementSystem = (ManagementSystem) management;
+IndexType indexType =
managementSystem.getSchemaVertex(index).asIndexType();
+if (!(indexType instanceof MixedIndexType)) {
+
[atlas] branch branch-2.0 updated: ATLAS-3977: Deleted entity behavior fix during migration. Part 2
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 374e20e ATLAS-3977: Deleted entity behavior fix during migration.
Part 2
374e20e is described below
commit 374e20e49931040bee996a56e32d477ceb049893
Author: Ashutosh Mestry
AuthorDate: Tue Oct 13 16:46:03 2020 -0700
ATLAS-3977: Deleted entity behavior fix during migration. Part 2
---
.../org/apache/atlas/repository/patches/UniqueAttributePatch.java| 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
b/repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
index d3111f1..c30bd25 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
@@ -18,6 +18,7 @@
package org.apache.atlas.repository.patches;
import org.apache.atlas.exception.AtlasBaseException;
+import org.apache.atlas.model.instance.AtlasEntity;
import org.apache.atlas.model.typedef.AtlasStructDef.AtlasAttributeDef;
import org.apache.atlas.pc.WorkItemManager;
import org.apache.atlas.repository.Constants;
@@ -164,7 +165,9 @@ public class UniqueAttributePatch extends AtlasPatchHandler
{
LOG.debug("processItem(typeName={}, vertexId={})", typeName,
vertexId);
processIndexStringAttribute(vertexId, vertex, typeName,
entityType);
-processUniqueAttribute(vertexId, vertex, typeName, entityType);
+if (AtlasGraphUtilsV2.getState(vertex) ==
AtlasEntity.Status.ACTIVE) {
+processUniqueAttribute(vertexId, vertex, typeName, entityType);
+}
LOG.debug("processItem(typeName={}, vertexId={}): Done!",
typeName, vertexId);
}
[atlas] branch master updated: ATLAS-3977: Deleted entity behavior fix during migration. Part 2
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 080e484 ATLAS-3977: Deleted entity behavior fix during migration.
Part 2
080e484 is described below
commit 080e4843480743558bc4b61d9c2860b414786ccc
Author: Ashutosh Mestry
AuthorDate: Tue Oct 13 16:46:03 2020 -0700
ATLAS-3977: Deleted entity behavior fix during migration. Part 2
---
.../org/apache/atlas/repository/patches/UniqueAttributePatch.java| 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
b/repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
index d3111f1..c30bd25 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
@@ -18,6 +18,7 @@
package org.apache.atlas.repository.patches;
import org.apache.atlas.exception.AtlasBaseException;
+import org.apache.atlas.model.instance.AtlasEntity;
import org.apache.atlas.model.typedef.AtlasStructDef.AtlasAttributeDef;
import org.apache.atlas.pc.WorkItemManager;
import org.apache.atlas.repository.Constants;
@@ -164,7 +165,9 @@ public class UniqueAttributePatch extends AtlasPatchHandler
{
LOG.debug("processItem(typeName={}, vertexId={})", typeName,
vertexId);
processIndexStringAttribute(vertexId, vertex, typeName,
entityType);
-processUniqueAttribute(vertexId, vertex, typeName, entityType);
+if (AtlasGraphUtilsV2.getState(vertex) ==
AtlasEntity.Status.ACTIVE) {
+processUniqueAttribute(vertexId, vertex, typeName, entityType);
+}
LOG.debug("processItem(typeName={}, vertexId={}): Done!",
typeName, vertexId);
}
[atlas] branch branch-2.0 updated: ATLAS-3989: Updated Export-Import Audits Writer to use metadata.namespace.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 757282f ATLAS-3989: Updated Export-Import Audits Writer to use
metadata.namespace.
757282f is described below
commit 757282f23338fa00c2a5112b538d43818d2057d0
Author: Ashutosh Mestry
AuthorDate: Fri Oct 9 17:07:25 2020 -0700
ATLAS-3989: Updated Export-Import Audits Writer to use metadata.namespace.
---
common/src/main/java/org/apache/atlas/AtlasConstants.java | 1 +
.../java/org/apache/atlas/repository/impexp/AuditsWriter.java | 8 ++--
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/common/src/main/java/org/apache/atlas/AtlasConstants.java
b/common/src/main/java/org/apache/atlas/AtlasConstants.java
index 319eb57..ce0dbd0 100644
--- a/common/src/main/java/org/apache/atlas/AtlasConstants.java
+++ b/common/src/main/java/org/apache/atlas/AtlasConstants.java
@@ -26,6 +26,7 @@ public final class AtlasConstants {
}
public static final String CLUSTER_NAME_KEY =
"atlas.cluster.name";
+public static final String METADATA_NAMESPACE_KEY=
"atlas.metadata.namespace";
public static final String DEFAULT_CLUSTER_NAME = "primary";
public static final String SYSTEM_PROPERTY_APP_PORT =
"atlas.app.port";
public static final String ATLAS_REST_ADDRESS_KEY=
"atlas.rest.address";
diff --git
a/repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
b/repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
index 55990f7..c4de0ed 100644
---
a/repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
+++
b/repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
@@ -120,13 +120,17 @@ public class AuditsWriter {
}
public static String getCurrentClusterName() {
+String ret = StringUtils.EMPTY;
try {
-return
ApplicationProperties.get().getString(AtlasConstants.CLUSTER_NAME_KEY,
CLUSTER_NAME_DEFAULT);
+ret =
ApplicationProperties.get().getString(AtlasConstants.METADATA_NAMESPACE_KEY,
StringUtils.EMPTY);
+if (StringUtils.isEmpty(ret)) {
+ret =
ApplicationProperties.get().getString(AtlasConstants.CLUSTER_NAME_KEY,
CLUSTER_NAME_DEFAULT);
+}
} catch (AtlasException e) {
LOG.error("getCurrentClusterName", e);
}
-return StringUtils.EMPTY;
+return ret;
}
static String getServerNameFromFullName(String fullName) {
[atlas] branch master updated: ATLAS-3989: Updated Export-Import Audits Writer to use metadata.namespace.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 39ff496 ATLAS-3989: Updated Export-Import Audits Writer to use
metadata.namespace.
39ff496 is described below
commit 39ff496f129b64352e91b462c7882491a3929981
Author: Ashutosh Mestry
AuthorDate: Fri Oct 9 17:07:25 2020 -0700
ATLAS-3989: Updated Export-Import Audits Writer to use metadata.namespace.
---
common/src/main/java/org/apache/atlas/AtlasConstants.java | 1 +
.../java/org/apache/atlas/repository/impexp/AuditsWriter.java | 8 ++--
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/common/src/main/java/org/apache/atlas/AtlasConstants.java
b/common/src/main/java/org/apache/atlas/AtlasConstants.java
index 319eb57..ce0dbd0 100644
--- a/common/src/main/java/org/apache/atlas/AtlasConstants.java
+++ b/common/src/main/java/org/apache/atlas/AtlasConstants.java
@@ -26,6 +26,7 @@ public final class AtlasConstants {
}
public static final String CLUSTER_NAME_KEY =
"atlas.cluster.name";
+public static final String METADATA_NAMESPACE_KEY=
"atlas.metadata.namespace";
public static final String DEFAULT_CLUSTER_NAME = "primary";
public static final String SYSTEM_PROPERTY_APP_PORT =
"atlas.app.port";
public static final String ATLAS_REST_ADDRESS_KEY=
"atlas.rest.address";
diff --git
a/repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
b/repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
index 55990f7..c4de0ed 100644
---
a/repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
+++
b/repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
@@ -120,13 +120,17 @@ public class AuditsWriter {
}
public static String getCurrentClusterName() {
+String ret = StringUtils.EMPTY;
try {
-return
ApplicationProperties.get().getString(AtlasConstants.CLUSTER_NAME_KEY,
CLUSTER_NAME_DEFAULT);
+ret =
ApplicationProperties.get().getString(AtlasConstants.METADATA_NAMESPACE_KEY,
StringUtils.EMPTY);
+if (StringUtils.isEmpty(ret)) {
+ret =
ApplicationProperties.get().getString(AtlasConstants.CLUSTER_NAME_KEY,
CLUSTER_NAME_DEFAULT);
+}
} catch (AtlasException e) {
LOG.error("getCurrentClusterName", e);
}
-return StringUtils.EMPTY;
+return ret;
}
static String getServerNameFromFullName(String fullName) {
[atlas] branch branch-2.0 updated: ATLAS-3987: Atlas client export API, must pass server error code in the exception
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 47c3155 ATLAS-3987: Atlas client export API, must pass server error
code in the exception
47c3155 is described below
commit 47c3155e04a438f02e2fd49740065ce983aa66e9
Author: Deep Singh
AuthorDate: Fri Oct 9 00:52:41 2020 -0700
ATLAS-3987: Atlas client export API, must pass server error code in the
exception
Signed-off-by: Ashutosh Mestry
---
.../src/main/java/org/apache/atlas/AtlasBaseClient.java | 6 +++---
.../main/java/org/apache/atlas/AtlasServiceException.java| 5 +
.../apache/atlas/web/resources/AdminExportImportTestIT.java | 12
3 files changed, 20 insertions(+), 3 deletions(-)
diff --git a/client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
b/client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
index 6e03669..cb35c94 100644
--- a/client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
+++ b/client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
@@ -477,16 +477,16 @@ public abstract class AtlasBaseClient {
public InputStream exportData(AtlasExportRequest request) throws
AtlasServiceException {
try {
return (InputStream) callAPI(EXPORT, Object.class, request);
-} catch (Exception e) {
-LOG.error("error writing to file", e);
+} catch (AtlasServiceException e) {
+LOG.error("error in export API call", e);
throw new AtlasServiceException(e);
}
}
public void exportData(AtlasExportRequest request, String absolutePath)
throws AtlasServiceException {
OutputStream fileOutputStream = null;
+InputStream inputStream = exportData(request);
try {
-InputStream inputStream = exportData(request);
fileOutputStream = new FileOutputStream(new File(absolutePath));
byte[] buffer = new byte[8 * 1024];
int bytesRead;
diff --git
a/client/common/src/main/java/org/apache/atlas/AtlasServiceException.java
b/client/common/src/main/java/org/apache/atlas/AtlasServiceException.java
index 6b4dd11..b9fe24a 100755
--- a/client/common/src/main/java/org/apache/atlas/AtlasServiceException.java
+++ b/client/common/src/main/java/org/apache/atlas/AtlasServiceException.java
@@ -41,6 +41,11 @@ public class AtlasServiceException extends Exception {
super(e);
}
+public AtlasServiceException(AtlasServiceException e) {
+super(e);
+this.status = e.status;
+}
+
public ClientResponse.Status getStatus() {
return status;
}
diff --git
a/webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java
b/webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java
index e53f9a8..51580af 100644
---
a/webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java
+++
b/webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java
@@ -19,6 +19,7 @@
package org.apache.atlas.web.resources;
+import org.apache.atlas.AtlasClientV2;
import org.apache.atlas.AtlasServiceException;
import org.apache.atlas.exception.AtlasBaseException;
import org.apache.atlas.model.impexp.AtlasExportResult;
@@ -84,6 +85,17 @@ public class AdminExportImportTestIT extends BaseResourceIT {
assertTrue(zs.getCreationOrder().size() >=
EXPECTED_CREATION_ORDER_SIZE, "expected creationOrderSize > " +
EXPECTED_CREATION_ORDER_SIZE + ", but found " + zs.getCreationOrder().size());
}
+@Test
+public void unAuthExportData() throws IOException {
+AtlasClientV2 unAuthClient = new AtlasClientV2(atlasUrls, new
String[]{"admin", "wr0ng_pa55w0rd"});
+AtlasExportRequest request =
TestResourceFileUtils.readObjectFromJson(".", EXPORT_REQUEST_FILE,
AtlasExportRequest.class);
+try {
+InputStream exportedStream = unAuthClient.exportData(request);
+} catch(AtlasServiceException e) {
+assertNotNull(e.getStatus(), "expected server error code in the
status");
+}
+}
+
private void performImport(String fileToImport, int
expectedProcessedEntitiesCount) throws AtlasServiceException {
AtlasImportRequest request = new AtlasImportRequest();
request.getOptions().put(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM,
SOURCE_SERVER_NAME);
[atlas] branch master updated: ATLAS-3987: Atlas client export API, must pass server error code in the exception
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new f5db98b ATLAS-3987: Atlas client export API, must pass server error
code in the exception
f5db98b is described below
commit f5db98b3ba5a6bdf851b1319abab5f953d6ff0f4
Author: Deep Singh
AuthorDate: Fri Oct 9 00:52:41 2020 -0700
ATLAS-3987: Atlas client export API, must pass server error code in the
exception
Signed-off-by: Ashutosh Mestry
---
.../src/main/java/org/apache/atlas/AtlasBaseClient.java | 6 +++---
.../main/java/org/apache/atlas/AtlasServiceException.java| 5 +
.../apache/atlas/web/resources/AdminExportImportTestIT.java | 12
3 files changed, 20 insertions(+), 3 deletions(-)
diff --git a/client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
b/client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
index 6e03669..cb35c94 100644
--- a/client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
+++ b/client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java
@@ -477,16 +477,16 @@ public abstract class AtlasBaseClient {
public InputStream exportData(AtlasExportRequest request) throws
AtlasServiceException {
try {
return (InputStream) callAPI(EXPORT, Object.class, request);
-} catch (Exception e) {
-LOG.error("error writing to file", e);
+} catch (AtlasServiceException e) {
+LOG.error("error in export API call", e);
throw new AtlasServiceException(e);
}
}
public void exportData(AtlasExportRequest request, String absolutePath)
throws AtlasServiceException {
OutputStream fileOutputStream = null;
+InputStream inputStream = exportData(request);
try {
-InputStream inputStream = exportData(request);
fileOutputStream = new FileOutputStream(new File(absolutePath));
byte[] buffer = new byte[8 * 1024];
int bytesRead;
diff --git
a/client/common/src/main/java/org/apache/atlas/AtlasServiceException.java
b/client/common/src/main/java/org/apache/atlas/AtlasServiceException.java
index 6b4dd11..b9fe24a 100755
--- a/client/common/src/main/java/org/apache/atlas/AtlasServiceException.java
+++ b/client/common/src/main/java/org/apache/atlas/AtlasServiceException.java
@@ -41,6 +41,11 @@ public class AtlasServiceException extends Exception {
super(e);
}
+public AtlasServiceException(AtlasServiceException e) {
+super(e);
+this.status = e.status;
+}
+
public ClientResponse.Status getStatus() {
return status;
}
diff --git
a/webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java
b/webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java
index e53f9a8..51580af 100644
---
a/webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java
+++
b/webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java
@@ -19,6 +19,7 @@
package org.apache.atlas.web.resources;
+import org.apache.atlas.AtlasClientV2;
import org.apache.atlas.AtlasServiceException;
import org.apache.atlas.exception.AtlasBaseException;
import org.apache.atlas.model.impexp.AtlasExportResult;
@@ -84,6 +85,17 @@ public class AdminExportImportTestIT extends BaseResourceIT {
assertTrue(zs.getCreationOrder().size() >=
EXPECTED_CREATION_ORDER_SIZE, "expected creationOrderSize > " +
EXPECTED_CREATION_ORDER_SIZE + ", but found " + zs.getCreationOrder().size());
}
+@Test
+public void unAuthExportData() throws IOException {
+AtlasClientV2 unAuthClient = new AtlasClientV2(atlasUrls, new
String[]{"admin", "wr0ng_pa55w0rd"});
+AtlasExportRequest request =
TestResourceFileUtils.readObjectFromJson(".", EXPORT_REQUEST_FILE,
AtlasExportRequest.class);
+try {
+InputStream exportedStream = unAuthClient.exportData(request);
+} catch(AtlasServiceException e) {
+assertNotNull(e.getStatus(), "expected server error code in the
status");
+}
+}
+
private void performImport(String fileToImport, int
expectedProcessedEntitiesCount) throws AtlasServiceException {
AtlasImportRequest request = new AtlasImportRequest();
request.getOptions().put(AtlasImportRequest.OPTION_KEY_REPLICATED_FROM,
SOURCE_SERVER_NAME);
[atlas] branch branch-2.0 updated: ATLAS-3977: Patch handler for addressing deleted entities after migration.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 2101d82 ATLAS-3977: Patch handler for addressing deleted entities
after migration.
2101d82 is described below
commit 2101d82b05d5aaf03b9e47e910f39777013325f0
Author: Ashutosh Mestry
AuthorDate: Wed Oct 7 21:30:17 2020 -0700
ATLAS-3977: Patch handler for addressing deleted entities after migration.
---
.../apache/atlas/repository/patches/ClassificationTextPatch.java| 6 ++
.../apache/atlas/repository/patches/ConcurrentPatchProcessor.java | 4
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/ClassificationTextPatch.java
b/repository/src/main/java/org/apache/atlas/repository/patches/ClassificationTextPatch.java
index 8351942..a3415c0 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/ClassificationTextPatch.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/ClassificationTextPatch.java
@@ -18,12 +18,14 @@
package org.apache.atlas.repository.patches;
import org.apache.atlas.exception.AtlasBaseException;
+import org.apache.atlas.model.instance.AtlasEntity;
import org.apache.atlas.pc.WorkItemManager;
import org.apache.atlas.repository.Constants;
import org.apache.atlas.repository.graphdb.AtlasEdge;
import org.apache.atlas.repository.graphdb.AtlasEdgeDirection;
import org.apache.atlas.repository.graphdb.AtlasGraph;
import org.apache.atlas.repository.graphdb.AtlasVertex;
+import org.apache.atlas.repository.store.graph.v2.AtlasGraphUtilsV2;
import org.apache.atlas.type.AtlasClassificationType;
import org.apache.atlas.type.AtlasEntityType;
import org.apache.atlas.type.AtlasTypeRegistry;
@@ -120,6 +122,10 @@ public class ClassificationTextPatch extends
AtlasPatchHandler {
LOG.debug("processItem(typeName={}, vertexId={})", typeName,
vertexId);
}
+if (AtlasGraphUtilsV2.getState(vertex) !=
AtlasEntity.Status.ACTIVE) {
+return;
+}
+
getEntityGraphMapper().updateClassificationTextAndNames(vertex);
if(LOG.isDebugEnabled()) {
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
b/repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
index 5a9ac2a..c6f0e64 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
@@ -204,10 +204,6 @@ public abstract class ConcurrentPatchProcessor {
return;
}
-if (AtlasGraphUtilsV2.getState(vertex) !=
AtlasEntity.Status.ACTIVE) {
-return;
-}
-
String typeName = AtlasGraphUtilsV2.getTypeName(vertex);
AtlasEntityType entityType =
typeRegistry.getEntityTypeByName(typeName);
if (entityType == null) {
[atlas] branch branch-2.0 updated: ATLAS-3965: Relationships with null values are missing in entity relationshipAttributes
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 6cc3cf6 ATLAS-3965: Relationships with null values are missing in
entity relationshipAttributes
6cc3cf6 is described below
commit 6cc3cf6b434e719b79434a01ea6eedaf3a3c0afd
Author: Deep Singh
AuthorDate: Wed Sep 30 14:17:59 2020 -0700
ATLAS-3965: Relationships with null values are missing in entity
relationshipAttributes
Signed-off-by: Ashutosh Mestry
---
.../atlas/repository/store/graph/v2/EntityGraphRetriever.java| 9 -
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
index 00146b1..2199277 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
@@ -1352,12 +1352,11 @@ public class EntityGraphRetriever {
break;
}
-if (ret != null) {
-entity.setRelationshipAttribute(attributeName, ret);
+// Set Relationship attributes, even if the value is null
+entity.setRelationshipAttribute(attributeName, ret);
-if (attributeEndDef.getIsLegacyAttribute() &&
!entity.hasAttribute(attributeName)) {
-entity.setAttribute(attributeName, toLegacyAttribute(ret));
-}
+if (attributeEndDef.getIsLegacyAttribute() &&
!entity.hasAttribute(attributeName)) {
+entity.setAttribute(attributeName, toLegacyAttribute(ret));
}
return ret;
[atlas] branch master updated: ATLAS-3977: Patch handler for addressing deleted entities after migration.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 54bab3f ATLAS-3977: Patch handler for addressing deleted entities
after migration.
54bab3f is described below
commit 54bab3f88ed580e54074fc0f26b01f6d103fde60
Author: Ashutosh Mestry
AuthorDate: Wed Oct 7 21:30:17 2020 -0700
ATLAS-3977: Patch handler for addressing deleted entities after migration.
---
.../apache/atlas/repository/patches/ClassificationTextPatch.java| 6 ++
.../apache/atlas/repository/patches/ConcurrentPatchProcessor.java | 4
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/ClassificationTextPatch.java
b/repository/src/main/java/org/apache/atlas/repository/patches/ClassificationTextPatch.java
index 8351942..a3415c0 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/ClassificationTextPatch.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/ClassificationTextPatch.java
@@ -18,12 +18,14 @@
package org.apache.atlas.repository.patches;
import org.apache.atlas.exception.AtlasBaseException;
+import org.apache.atlas.model.instance.AtlasEntity;
import org.apache.atlas.pc.WorkItemManager;
import org.apache.atlas.repository.Constants;
import org.apache.atlas.repository.graphdb.AtlasEdge;
import org.apache.atlas.repository.graphdb.AtlasEdgeDirection;
import org.apache.atlas.repository.graphdb.AtlasGraph;
import org.apache.atlas.repository.graphdb.AtlasVertex;
+import org.apache.atlas.repository.store.graph.v2.AtlasGraphUtilsV2;
import org.apache.atlas.type.AtlasClassificationType;
import org.apache.atlas.type.AtlasEntityType;
import org.apache.atlas.type.AtlasTypeRegistry;
@@ -120,6 +122,10 @@ public class ClassificationTextPatch extends
AtlasPatchHandler {
LOG.debug("processItem(typeName={}, vertexId={})", typeName,
vertexId);
}
+if (AtlasGraphUtilsV2.getState(vertex) !=
AtlasEntity.Status.ACTIVE) {
+return;
+}
+
getEntityGraphMapper().updateClassificationTextAndNames(vertex);
if(LOG.isDebugEnabled()) {
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
b/repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
index 5a9ac2a..c6f0e64 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/ConcurrentPatchProcessor.java
@@ -204,10 +204,6 @@ public abstract class ConcurrentPatchProcessor {
return;
}
-if (AtlasGraphUtilsV2.getState(vertex) !=
AtlasEntity.Status.ACTIVE) {
-return;
-}
-
String typeName = AtlasGraphUtilsV2.getTypeName(vertex);
AtlasEntityType entityType =
typeRegistry.getEntityTypeByName(typeName);
if (entityType == null) {
[atlas] branch master updated: ATLAS-3965: Relationships with null values are missing in entity relationshipAttributes
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 877e2b1 ATLAS-3965: Relationships with null values are missing in
entity relationshipAttributes
877e2b1 is described below
commit 877e2b1729f3628d844c32769c30a3c948d567a6
Author: Deep Singh
AuthorDate: Wed Sep 30 14:17:59 2020 -0700
ATLAS-3965: Relationships with null values are missing in entity
relationshipAttributes
Signed-off-by: Ashutosh Mestry
---
.../atlas/repository/store/graph/v2/EntityGraphRetriever.java| 9 -
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
index 00146b1..2199277 100644
---
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
+++
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
@@ -1352,12 +1352,11 @@ public class EntityGraphRetriever {
break;
}
-if (ret != null) {
-entity.setRelationshipAttribute(attributeName, ret);
+// Set Relationship attributes, even if the value is null
+entity.setRelationshipAttribute(attributeName, ret);
-if (attributeEndDef.getIsLegacyAttribute() &&
!entity.hasAttribute(attributeName)) {
-entity.setAttribute(attributeName, toLegacyAttribute(ret));
-}
+if (attributeEndDef.getIsLegacyAttribute() &&
!entity.hasAttribute(attributeName)) {
+entity.setAttribute(attributeName, toLegacyAttribute(ret));
}
return ret;
[atlas] branch branch-2.0 updated: ATLAS-3953: Export: ZipSink: Specify character endcoding when writing to ZIP file.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 31b8bdb ATLAS-3953: Export: ZipSink: Specify character endcoding when
writing to ZIP file.
31b8bdb is described below
commit 31b8bdb404978e46c5061a6a8423ed8fbb6d3893
Author: Ashutosh Mestry
AuthorDate: Wed Sep 23 10:54:16 2020 -0700
ATLAS-3953: Export: ZipSink: Specify character endcoding when writing to
ZIP file.
Change-Id: I2e3a2bad79f5cf4c91328ed9f64a454ded2acbf3
---
.../repository/impexp/HdfsPathEntityCreator.java | 2 +-
.../apache/atlas/repository/impexp/ZipSink.java| 22 ++
.../atlas/repository/impexp/ZipSinkTest.java | 47 ++
3 files changed, 8 insertions(+), 63 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreator.java
b/repository/src/main/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreator.java
index 4a09c0f..679a577 100644
---
a/repository/src/main/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreator.java
+++
b/repository/src/main/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreator.java
@@ -59,7 +59,7 @@ public class HdfsPathEntityCreator {
}
public AtlasEntity.AtlasEntityWithExtInfo getCreateEntity(AtlasObjectId
item) throws AtlasBaseException {
-
if(!item.getUniqueAttributes().containsKey(HDFS_PATH_ATTRIBUTE_NAME_PATH)) {
+if(item.getUniqueAttributes() == null ||
!item.getUniqueAttributes().containsKey(HDFS_PATH_ATTRIBUTE_NAME_PATH)) {
return null;
}
diff --git
a/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java
b/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java
index 6375454..5cec39d 100644
--- a/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java
+++ b/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java
@@ -22,12 +22,13 @@ import org.apache.atlas.model.impexp.AtlasExportResult;
import org.apache.atlas.model.instance.AtlasEntity;
import org.apache.atlas.model.typedef.AtlasTypesDef;
import org.apache.atlas.type.AtlasType;
-import org.apache.commons.lang.StringUtils;
+import org.apache.commons.io.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
@@ -109,24 +110,7 @@ public class ZipSink {
}
private void writeBytes(String payload) throws IOException {
-splitAndWriteBytes(payload, 10 * 1024 * 1024, zipOutputStream);
-}
-
-static void splitAndWriteBytes(String msg, int bufferSize, OutputStream
os) throws IOException {
-int numberOfSplits = (int) Math.ceil(((float) msg.length()) /
bufferSize);
-if (numberOfSplits == 0) {
-numberOfSplits = 1;
-} else {
-if (LOG.isDebugEnabled()) {
-LOG.info("ZipSink: number of splits: {}", numberOfSplits);
-}
-}
-
-for (int i = 0, start = 0; i < numberOfSplits; i++, start +=
bufferSize) {
-int end = bufferSize + start;
-String s = StringUtils.substring(msg, start, end);
-os.write(s.getBytes());
-}
+IOUtils.copy(IOUtils.toInputStream(payload, StandardCharsets.UTF_8),
zipOutputStream);
}
public boolean hasEntity(String guid) {
diff --git
a/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSinkTest.java
b/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSinkTest.java
index cf6d16b..827d72f 100644
---
a/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSinkTest.java
+++
b/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSinkTest.java
@@ -30,15 +30,16 @@ import org.testng.annotations.Test;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
-import java.io.OutputStream;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
-import static org.mockito.Mockito.mock;
-import static org.testng.Assert.*;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
public class ZipSinkTest {
private ByteArrayOutputStream byteArrayOutputStream = new
ByteArrayOutputStream();
@@ -47,23 +48,6 @@ public class ZipSinkTest {
private AtlasExportResult defaultExportResult;
private String knownEntityGuidFormat = "111-222-333-%s";
-private class MockOutputStream extends OutputStream {
[atlas] branch master updated: ATLAS-3953: Export: ZipSink: Specify character endcoding when writing to ZIP file.
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new 9a605b6 ATLAS-3953: Export: ZipSink: Specify character endcoding when
writing to ZIP file.
9a605b6 is described below
commit 9a605b6671b1719eebce04605f100e34c8f5afb0
Author: Ashutosh Mestry
AuthorDate: Wed Sep 23 10:54:16 2020 -0700
ATLAS-3953: Export: ZipSink: Specify character endcoding when writing to
ZIP file.
Change-Id: I2e3a2bad79f5cf4c91328ed9f64a454ded2acbf3
---
.../repository/impexp/HdfsPathEntityCreator.java | 2 +-
.../apache/atlas/repository/impexp/ZipSink.java| 22 ++
.../atlas/repository/impexp/ZipSinkTest.java | 47 ++
3 files changed, 8 insertions(+), 63 deletions(-)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreator.java
b/repository/src/main/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreator.java
index 4a09c0f..679a577 100644
---
a/repository/src/main/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreator.java
+++
b/repository/src/main/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreator.java
@@ -59,7 +59,7 @@ public class HdfsPathEntityCreator {
}
public AtlasEntity.AtlasEntityWithExtInfo getCreateEntity(AtlasObjectId
item) throws AtlasBaseException {
-
if(!item.getUniqueAttributes().containsKey(HDFS_PATH_ATTRIBUTE_NAME_PATH)) {
+if(item.getUniqueAttributes() == null ||
!item.getUniqueAttributes().containsKey(HDFS_PATH_ATTRIBUTE_NAME_PATH)) {
return null;
}
diff --git
a/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java
b/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java
index 6375454..5cec39d 100644
--- a/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java
+++ b/repository/src/main/java/org/apache/atlas/repository/impexp/ZipSink.java
@@ -22,12 +22,13 @@ import org.apache.atlas.model.impexp.AtlasExportResult;
import org.apache.atlas.model.instance.AtlasEntity;
import org.apache.atlas.model.typedef.AtlasTypesDef;
import org.apache.atlas.type.AtlasType;
-import org.apache.commons.lang.StringUtils;
+import org.apache.commons.io.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
@@ -109,24 +110,7 @@ public class ZipSink {
}
private void writeBytes(String payload) throws IOException {
-splitAndWriteBytes(payload, 10 * 1024 * 1024, zipOutputStream);
-}
-
-static void splitAndWriteBytes(String msg, int bufferSize, OutputStream
os) throws IOException {
-int numberOfSplits = (int) Math.ceil(((float) msg.length()) /
bufferSize);
-if (numberOfSplits == 0) {
-numberOfSplits = 1;
-} else {
-if (LOG.isDebugEnabled()) {
-LOG.info("ZipSink: number of splits: {}", numberOfSplits);
-}
-}
-
-for (int i = 0, start = 0; i < numberOfSplits; i++, start +=
bufferSize) {
-int end = bufferSize + start;
-String s = StringUtils.substring(msg, start, end);
-os.write(s.getBytes());
-}
+IOUtils.copy(IOUtils.toInputStream(payload, StandardCharsets.UTF_8),
zipOutputStream);
}
public boolean hasEntity(String guid) {
diff --git
a/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSinkTest.java
b/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSinkTest.java
index cf6d16b..827d72f 100644
---
a/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSinkTest.java
+++
b/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSinkTest.java
@@ -30,15 +30,16 @@ import org.testng.annotations.Test;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
-import java.io.OutputStream;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
-import static org.mockito.Mockito.mock;
-import static org.testng.Assert.*;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
public class ZipSinkTest {
private ByteArrayOutputStream byteArrayOutputStream = new
ByteArrayOutputStream();
@@ -47,23 +48,6 @@ public class ZipSinkTest {
private AtlasExportResult defaultExportResult;
private String knownEntityGuidFormat = "111-222-333-%s";
-private class MockOutputStream extends OutputStream {
-
[atlas] branch branch-2.0 updated: ATLAS-3948: Entity Creation: Index Consistency: Java Patch Handler: Provide Option to Disable
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new a2ea7d3 ATLAS-3948: Entity Creation: Index Consistency: Java Patch
Handler: Provide Option to Disable
a2ea7d3 is described below
commit a2ea7d302e1ef8d6e9dd6f88e25749233a40e0f6
Author: Ashutosh Mestry
AuthorDate: Thu Sep 17 13:09:46 2020 -0700
ATLAS-3948: Entity Creation: Index Consistency: Java Patch Handler: Provide
Option to Disable
---
.../org/apache/atlas/repository/patches/IndexConsistencyPatch.java | 6 ++
1 file changed, 6 insertions(+)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
b/repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
index 2abe73c..9549e2d 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
@@ -17,6 +17,7 @@
*/
package org.apache.atlas.repository.patches;
+import org.apache.atlas.AtlasConfiguration;
import org.apache.atlas.exception.AtlasBaseException;
import org.apache.atlas.repository.graphdb.AtlasGraph;
import org.apache.atlas.type.AtlasTypeRegistry;
@@ -40,6 +41,11 @@ public class IndexConsistencyPatch extends AtlasPatchHandler
{
@Override
public void apply() throws AtlasBaseException {
+if (AtlasConfiguration.STORAGE_CONSISTENCY_LOCK_ENABLED.getBoolean()
== false) {
+LOG.info("IndexConsistencyPatch: Not enabled: Skipped!");
+return;
+}
+
AtlasGraph graph = context.getGraph();
try {
[atlas] branch master updated: ATLAS-3948: Entity Creation: Index Consistency: Java Patch Handler: Provide Option to Disable
This is an automated email from the ASF dual-hosted git repository.
amestry pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git
The following commit(s) were added to refs/heads/master by this push:
new eccc376 ATLAS-3948: Entity Creation: Index Consistency: Java Patch
Handler: Provide Option to Disable
eccc376 is described below
commit eccc37623e7c3d43e1aa494f88a5ad155fe9c123
Author: Ashutosh Mestry
AuthorDate: Thu Sep 17 13:09:46 2020 -0700
ATLAS-3948: Entity Creation: Index Consistency: Java Patch Handler: Provide
Option to Disable
---
.../org/apache/atlas/repository/patches/IndexConsistencyPatch.java | 6 ++
1 file changed, 6 insertions(+)
diff --git
a/repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
b/repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
index 2abe73c..9549e2d 100644
---
a/repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
+++
b/repository/src/main/java/org/apache/atlas/repository/patches/IndexConsistencyPatch.java
@@ -17,6 +17,7 @@
*/
package org.apache.atlas.repository.patches;
+import org.apache.atlas.AtlasConfiguration;
import org.apache.atlas.exception.AtlasBaseException;
import org.apache.atlas.repository.graphdb.AtlasGraph;
import org.apache.atlas.type.AtlasTypeRegistry;
@@ -40,6 +41,11 @@ public class IndexConsistencyPatch extends AtlasPatchHandler
{
@Override
public void apply() throws AtlasBaseException {
+if (AtlasConfiguration.STORAGE_CONSISTENCY_LOCK_ENABLED.getBoolean()
== false) {
+LOG.info("IndexConsistencyPatch: Not enabled: Skipped!");
+return;
+}
+
AtlasGraph graph = context.getGraph();
try {
