[jira] [Updated] (ATLAS-2812) Upgrade commons-fileupload to 1.3.3

[Sarath Subramanian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-2812:
--
Attachment: 0001-ATLAS-2812-Upgrade-commons-fileupload-to-1.3.3-master.patch

> Upgrade commons-fileupload to 1.3.3
> ---
>
> Key: ATLAS-2812
> URL: https://issues.apache.org/jira/browse/ATLAS-2812
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 0.8-incubating, 0.8.1, 0.8.2
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0, 0.8.3, 1.1.0
>
> Attachments: 
> 0001-ATLAS-2812-Upgrade-commons-fileupload-to-1.3.3-master.patch, 
> 0001-ATLAS-2812-Upgrade-commons-fileupload-to-1.3.3.patch
>
>
> Atlas packages commons-fileupload-1.2.1.jar which has a potential security 
> vulnerability
> Need to upgrade to 1.3.1+.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-2812) Upgrade commons-fileupload to 1.3.3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576879#comment-16576879
 ] 

ASF subversion and git services commented on ATLAS-2812:


Commit e0ac86961146af217696493f170b4b20836cd6ce in atlas's branch 
refs/heads/master from [~sarath.ku...@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=atlas.git;h=e0ac869 ]

ATLAS-2812: Upgrade commons-fileupload to 1.3.3


> Upgrade commons-fileupload to 1.3.3
> ---
>
> Key: ATLAS-2812
> URL: https://issues.apache.org/jira/browse/ATLAS-2812
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 0.8-incubating, 0.8.1, 0.8.2
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0, 0.8.3, 1.1.0
>
> Attachments: 
> 0001-ATLAS-2812-Upgrade-commons-fileupload-to-1.3.3-master.patch, 
> 0001-ATLAS-2812-Upgrade-commons-fileupload-to-1.3.3.patch
>
>
> Atlas packages commons-fileupload-1.2.1.jar which has a potential security 
> vulnerability
> Need to upgrade to 1.3.1+.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-2812) Upgrade commons-fileupload to 1.3.3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576889#comment-16576889
 ] 

ASF subversion and git services commented on ATLAS-2812:


Commit 0212b56a400283f96028b77e9b5526d8c23b147d in atlas's branch 
refs/heads/branch-1.0 from [~sarath.ku...@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=atlas.git;h=0212b56 ]

ATLAS-2812: Upgrade commons-fileupload to 1.3.3

(cherry picked from commit e0ac86961146af217696493f170b4b20836cd6ce)


> Upgrade commons-fileupload to 1.3.3
> ---
>
> Key: ATLAS-2812
> URL: https://issues.apache.org/jira/browse/ATLAS-2812
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 0.8-incubating, 0.8.1, 0.8.2
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0, 0.8.3, 1.1.0
>
> Attachments: 
> 0001-ATLAS-2812-Upgrade-commons-fileupload-to-1.3.3-master.patch, 
> 0001-ATLAS-2812-Upgrade-commons-fileupload-to-1.3.3.patch
>
>
> Atlas packages commons-fileupload-1.2.1.jar which has a potential security 
> vulnerability
> Need to upgrade to 1.3.1+.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: Atlas-1.0-IntegrationTests #736

[Apache Jenkins Server]

See 


--
[...truncated 760.39 KB...]
at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute 
(JettyDeployWar.java:65)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:208)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main 
(Launcher.java:356)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0 (Native Method)
at sun.nio.ch.Net.bind (Net.java:433)
at sun.nio.ch.Net.bind (Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind 
(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind (ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind (ServerSocketAdaptor.java:67)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure 
(NIOServerCnxnFactory.java:95)
at org.apache.zookeeper.server.ServerCnxnFactory.createFactory 
(ServerCnxnFactory.java:126)
at org.apache.atlas.kafka.EmbeddedKafkaServer.startZk 
(EmbeddedKafkaServer.java:116)
at org.apache.atlas.kafka.EmbeddedKafkaServer.start 
(EmbeddedKafkaServer.java:80)
at org.apache.atlas.service.Services.start (Services.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at 
org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke
 (InitDestroyAnnotationBeanPostProcessor.java:366)
at 
org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods
 (InitDestroyAnnotationBeanPostProcessor.java:311)
at 
org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization
 (InitDestroyAnnotationBeanPostProcessor.java:134)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization
 (AbstractAutowireCapableBeanFactory.java:409)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean
 (AbstractAutowireCapableBeanFactory.java:1626)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean
 (AbstractAutowireCapableBeanFactory.java:555)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean
 (AbstractAutowireCapableBeanFactory.java:483)
at 
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject 
(AbstractBeanFactory.java:312)
at 
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton
 (DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean 
(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean 
(AbstractBeanFactory.java:197)
at 
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons
 

Re: Review Request 68287: ATLAS-2807: Re-evaluate classification propagation during entity delete

[Sarath Subramanian]

> On Aug. 10, 2018, 3:37 a.m., Graham Wallis wrote:
> > Please look at the question I posted to the JIRA - I am not sure that you 
> > always want the specified behaviour.

The behavior doesn't depend on the nature of the relationship, although the 
example showcases retaining propagated tags for Dataset -> Process -> Dataset 
relationship. This behavior is applicable to all relationship types. We are 
planning to add a global property to toggle the default behavior.

for e.g. if retainPropagatedClassificationsOnEntityDelete = true/false. This 
will be implemented in a follow up JIRA.


- Sarath


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68287/#review207065
---


On Aug. 9, 2018, 7:36 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68287/
> ---
> 
> (Updated Aug. 9, 2018, 7:36 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and 
> Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-2807
> https://issues.apache.org/jira/browse/ATLAS-2807
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The current behavior - when we delete an entity f1, all tags associate to f1 
> which got propagated
> 
> 
> to downstream entities are removed – PII tag propagated to process1 and t1 is 
> removed
> 
> 
> Proposed Change
> 
> 
> Soft Delete:
> 
> 
> When entity f1 is deleted, retain the propagated classification edges, so the 
> downstream entities
> 
> 
> – process1 and t1 continue to have PII classification associated to them
> 
> 
> Hard Delete:
> 
> 
> In cases of hard delete, the source entity (f1) is deleted from atlas, but 
> its classification vertex
> 
> 
> (PII) continue to exist and will continue propagating to process1 and t1.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 7988d1d81 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 328b76734 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
> 3132379b9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> d328873f1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  c6be4061b 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  4a0924bbe 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
>  a95e6898e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java
>  41e65d4a3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  28a26ab4f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  f57ce99f7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  8f4faaff8 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  58005c458 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  8ceeedb01 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> e01cba92d 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java fd331faa0 
>   webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java 
> 78bd53ce7 
> 
> 
> Diff: https://reviews.apache.org/r/68287/diff/1/
> 
> 
> Testing
> ---
> 
> Validated basic cases using REST clirnt.
> 
> 
> PreCommit: In progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Build failed in Jenkins: Atlas-0.8-IntegrationTests #232

[Apache Jenkins Server]

See 


Changes:

[ssubramanian] ATLAS-2812: Upgrade commons-fileupload to 1.3.3

--
[...truncated 485.72 KB...]
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main 
(Launcher.java:356)
[WARNING] FAILED ServerConnector@3ff63000{HTTP/1.1}{0.0.0.0:31000}: 
java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0 (Native Method)
at sun.nio.ch.Net.bind (Net.java:433)
at sun.nio.ch.Net.bind (Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind 
(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind (ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.open (ServerConnector.java:321)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart 
(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart 
(ServerConnector.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
(AbstractLifeCycle.java:68)
at org.eclipse.jetty.maven.plugin.MavenServerConnector.doStart 
(MavenServerConnector.java:120)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart (Server.java:366)
at org.eclipse.jetty.maven.plugin.JettyServer.doStart (JettyServer.java:73)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start 
(AbstractLifeCycle.java:68)
at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.startJetty 
(AbstractJettyMojo.java:534)
at org.eclipse.jetty.maven.plugin.AbstractJettyMojo.execute 
(AbstractJettyMojo.java:357)
at org.eclipse.jetty.maven.plugin.JettyRunWarMojo.execute 
(JettyRunWarMojo.java:64)
at org.eclipse.jetty.maven.plugin.JettyDeployWar.execute 
(JettyDeployWar.java:67)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:208)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main 
(Launcher.java:356)
[WARNING] FAILED org.eclipse.jetty.maven.plugin.MavenServerConnector@28a6d54f: 
java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0 (Native Method)
at sun.nio.ch.Net.bind (Net.java:433)
at sun.nio.ch.Net.bind (Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind 
(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind (ServerSocketAdaptor.java:74)
at 

[jira] [Commented] (ATLAS-2812) Upgrade commons-fileupload to 1.3.3

[Apoorv Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576860#comment-16576860
 ] 

Apoorv Naik commented on ATLAS-2812:


Looks good for commit. Thanks Sarath.

> Upgrade commons-fileupload to 1.3.3
> ---
>
> Key: ATLAS-2812
> URL: https://issues.apache.org/jira/browse/ATLAS-2812
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 0.8-incubating, 0.8.1, 0.8.2
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0, 0.8.3, 1.1.0
>
> Attachments: 0001-ATLAS-2812-Upgrade-commons-fileupload-to-1.3.3.patch
>
>
> Atlas packages commons-fileupload-1.2.1.jar which has a potential security 
> vulnerability
> Need to upgrade to 1.3.1+.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (ATLAS-2812) Upgrade commons-fileupload to 1.3.3

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576863#comment-16576863
 ] 

ASF subversion and git services commented on ATLAS-2812:


Commit ac7c5e11d7de761ba8dba2499260f7f0d54ee409 in atlas's branch 
refs/heads/branch-0.8 from [~sarath.ku...@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=atlas.git;h=ac7c5e1 ]

ATLAS-2812: Upgrade commons-fileupload to 1.3.3


> Upgrade commons-fileupload to 1.3.3
> ---
>
> Key: ATLAS-2812
> URL: https://issues.apache.org/jira/browse/ATLAS-2812
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 0.8-incubating, 0.8.1, 0.8.2
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0, 0.8.3, 1.1.0
>
> Attachments: 0001-ATLAS-2812-Upgrade-commons-fileupload-to-1.3.3.patch
>
>
> Atlas packages commons-fileupload-1.2.1.jar which has a potential security 
> vulnerability
> Need to upgrade to 1.3.1+.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68270: Cluster Entity Updated after Successful Import if Replication Options are Available in Import Request

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68270/#review207097
---




intg/src/main/java/org/apache/atlas/model/impexp/ReplicationDetails.java
Lines 38 (patched)


consider making 'operation' an enum with values IMPORT/EXPORT



repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
Line 107 (original), 97 (patched)


AtlasCluster name and qualifiedName is same?


- Sarath Subramanian


On Aug. 9, 2018, 1:19 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68270/
> ---
> 
> (Updated Aug. 9, 2018, 1:19 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2814
> https://issues.apache.org/jira/browse/ATLAS-2814
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> - New model _ReplicationDetails_ store replication timestamp.
> - _AuditWriter_ updates appropriate _AtlasCluster_ entity with 
> _ReplicationDetails_.
> 
> 
> Diffs
> -
> 
>   client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java 
> f73ba2c6e6c9d05e86eb6c17f923a6dda1df5dd9 
>   intg/src/main/java/org/apache/atlas/model/clusterinfo/AtlasCluster.java 
> 3ce50e38b3af45521c2c83d0e6c05398747bcf86 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 
> 85a606c7d3b2158238b2a6defbb9185883434d78 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java 
> bfb7637711e08e17fd5f5b16bf121ac541b631a4 
>   intg/src/main/java/org/apache/atlas/model/impexp/ReplicationDetails.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java 
> 6a3fbecda84dc48e0df1359c1b5a248958db0a4c 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ClusterService.java
>  fd8e2bfe6bbdb7ae5e66ab67ecdf94e9527d9b09 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportImportAuditService.java
>  e90b6b942d359f78ccf59ee955c4667f2033e6c4 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
>  b15f828e470eaee972a4ee3f8447f404f813c699 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  98ef389bd1bf8f87d23a06dc231d4d93f06b2231 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasClusterDTO.java 
> 424fb88a5ad2d84ac2245848d061892d40a3f50c 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  9e7a119a3abfb50401f9747dad1ffa55facc51ed 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ClusterServiceTest.java
>  cfd272fb00a970ab0ba28f96b237de6bd5c694e1 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
>  f3803e5ae15a772eb4ff04192a83a5b412631791 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java
>  fcf90d364950d3afa2c18689d8e22616cfcd3c19 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
>  86ab22294918701270db3ba849f5b0532c1a21a4 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  dc25e92d7115db31cbd7f99bd1a2cb5f75e766a5 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java
>  881368ce2c6803c8cf887a1fc91737c045c5e663 
>   server-api/src/main/java/org/apache/atlas/RequestContextV1.java 
> 8506d186978f579c6b18c4347fc7cb2636848f16 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> d4e72620b15f69d258e5a2a96521c65a7ef71114 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/AdminExportImportTestIT.java
>  fc804d2e5a1debd7e728dca1773f91402d41fc26 
>   webapp/src/test/resources/json/export-incremental.json 
> 97108411f9e0c177ac13eb0a8ec3bd4475499f89 
>   webapp/src/test/resources/stocks-base.zip 
> 40c7f37eefb46a944921f6a74a916191704cb9a3 
> 
> 
> Diff: https://reviews.apache.org/r/68270/diff/3/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> New tests added.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 68287: ATLAS-2807: Re-evaluate classification propagation during entity delete

[Sarath Subramanian]

> On Aug. 10, 2018, 4:01 a.m., David Radley wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
> > Line 56 (original)
> > 
> >
> > I assume this is implementing the reporting text "Hard Delete:
> > In cases of hard delete, the source entity (f1) is deleted from atlas, 
> > but its classification vertex
> > (PII) continue to exist and will continue propagating to process1 and 
> > t1." 
> > 
> > I am not sure about this - as if a source entity is PII and it is 
> > deleted - I would think any downstream entities should not be PII.

David, we are planning to add a global property to toggle this default behavior.


for e.g
consider the following example where we assign PII to medical_records_table and 
add security policies in ranger enforcing authorized users to access these 
sensitive table and view. If the source table is deleted for some reason and if 
we remove the propagated classification, this might possibly pose a security 
access issue. Sensitive data is now accesible to everyone. This JIRA solves 
this case. 

medical_records_table -> Process -> uk_medical_records_table
[PII] {PII}{PII}


- Sarath


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68287/#review207066
---


On Aug. 9, 2018, 7:36 p.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68287/
> ---
> 
> (Updated Aug. 9, 2018, 7:36 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and 
> Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-2807
> https://issues.apache.org/jira/browse/ATLAS-2807
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The current behavior - when we delete an entity f1, all tags associate to f1 
> which got propagated
> 
> 
> to downstream entities are removed – PII tag propagated to process1 and t1 is 
> removed
> 
> 
> Proposed Change
> 
> 
> Soft Delete:
> 
> 
> When entity f1 is deleted, retain the propagated classification edges, so the 
> downstream entities
> 
> 
> – process1 and t1 continue to have PII classification associated to them
> 
> 
> Hard Delete:
> 
> 
> In cases of hard delete, the source entity (f1) is deleted from atlas, but 
> its classification vertex
> 
> 
> (PII) continue to exist and will continue propagating to process1 and t1.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 7988d1d81 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 328b76734 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
> 3132379b9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> d328873f1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  c6be4061b 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  4a0924bbe 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
>  a95e6898e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java
>  41e65d4a3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  28a26ab4f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  f57ce99f7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  8f4faaff8 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  58005c458 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  8ceeedb01 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> e01cba92d 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java fd331faa0 
>   webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java 
> 78bd53ce7 
> 
> 
> Diff: https://reviews.apache.org/r/68287/diff/1/
> 
> 
> Testing
> ---
> 
> Validated basic cases using REST clirnt.
> 
> 
> PreCommit: In progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Jenkins build is back to normal : Atlas-1.0-IntegrationTests #737

[Apache Jenkins Server]

See 

[jira] [Commented] (ATLAS-2807) Re-evaluate classification propagation during entity delete

[Sarath Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576751#comment-16576751
 ] 

Sarath Subramanian commented on ATLAS-2807:
---

[~grahamwallis], the behavior doesn't depend on the nature of the relationship, 
although the example showcases retaining propagated tags for Dataset -> Process 
-> Dataset relationship. This behavior is applicable to all relationship types. 
We are planning to add a global property to toggle the default behavior.

for e.g. if retainPropagatedClassificationsOnEntityDelete = true/false. This 
will be implemented in a follow up JIRA.

 

> Re-evaluate classification propagation during entity delete
> ---
>
> Key: ATLAS-2807
> URL: https://issues.apache.org/jira/browse/ATLAS-2807
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.0.0-alpha, 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.1.0
>
> Attachments: 
> ATLAS-2807-Re-evaluate-classification-propagation-du-MASTER.patch, 
> Classification Propagation during entity delete1.pdf
>
>
> The current behavior - when we delete an entity *f1*, all tags associate to 
> *f1* which got propagated
> to downstream entities are removed – *PII* tag propagated to *process1* and 
> *t1* is removed
> *Proposed Change*
> _*Soft Delete:*_
> When entity *f1* is deleted, retain the propagated classification edges, so 
> the downstream entities
> – *process1* and *t1* continue to have *PII* classification associated to them
> _*Hard Delete:*_
> In cases of hard delete, the source entity (*f1*) is deleted from atlas, but 
> its classification vertex
> (*PII*) continue to exist and will continue propagating to *process1* and 
> *t1*.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: Atlas-0.8-UnitTests #234

[Apache Jenkins Server]

See 


Changes:

[ssubramanian] ATLAS-2812: Upgrade commons-fileupload to 1.3.3

--
[...truncated 195.19 KB...]
Running org.apache.atlas.utils.TestResourceFileUtils
Running org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1Test
Running org.apache.atlas.repository.impexp.ExportImportAuditServiceTest
Running org.apache.atlas.repository.impexp.ReplicationEntityAttributeTest
Running org.apache.atlas.repository.impexp.ImportTransformsTest
Running org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1Test
Running org.apache.atlas.repository.store.graph.v1.HardDeleteHandlerV1Test
Running org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryTest
Running org.apache.atlas.repository.impexp.ImportTransformerJSONTest
Running org.apache.atlas.repository.impexp.TypeAttributeDifferenceTest
Running 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1BulkImportPercentTest
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateHardDeleteV1Test
Running org.apache.atlas.repository.userprofile.UserProfileServiceTest
Running org.apache.atlas.repository.typestore.GraphBackedTypeStoreTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.314 sec - in 
org.apache.atlas.repository.impexp.ImportTransformerJSONTest
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.247 sec - in 
org.apache.atlas.repository.BaseTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.339 sec - in 
org.apache.atlas.utils.TestResourceFileUtils
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.008 sec - in 
org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryHATest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.469 sec - in 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateV1Test
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Running 
org.apache.atlas.repository.store.graph.v1.InverseReferenceUpdateSoftDeleteV1Test
Running org.apache.atlas.repository.store.graph.v1.AtlasDeleteHandlerV1Test
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.177 sec - in 
org.apache.atlas.utils.ObjectUpdateSynchronizerTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.378 sec - in 
org.apache.atlas.repository.impexp.AtlasImportRequestTest
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.39 sec - in 
org.apache.atlas.repository.impexp.ImportTransformsTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.235 sec - in 
org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1BulkImportPercentTest
Running org.apache.atlas.repository.graph.GraphBackedRepositorySoftDeleteTest
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.154 sec - in 
org.apache.atlas.repository.impexp.TypeAttributeDifferenceTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.706 sec - in 
org.apache.atlas.repository.impexp.UniqueListTest
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Running org.apache.atlas.repository.graph.Gremlin2QueryOptimizerTest
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.515 sec - in 
org.apache.atlas.repository.impexp.ImportTransformerTest
Running org.apache.atlas.repository.graph.GraphBackedRepositoryHardDeleteTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.864 sec - in 
org.apache.atlas.repository.impexp.ZipSinkTest
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Running org.apache.atlas.repository.graph.GraphBackedMetadataRepositoryTest
Running org.apache.atlas.repository.graph.AbstractGremlinQueryOptimizerTest
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
Running 

Re: Review Request 68287: ATLAS-2807: Re-evaluate classification propagation during entity delete

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68287/#review207060
---


Fix it, then Ship it!





intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
Line 131 (original), 131 (patched)


"Propagated classification {0} - associatedEntityGuid {1} is not associated 
with entity {2}" ==>

"Propagated classification {0} is not associated with entity {2}, it is 
associated with entity {1}"



intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
Lines 155 (patched)


"Propagated classification {0} with associatedEntityGuid {1} removal not 
supported since associated entity is ACTIVE" ==>

"Removal of classification {0}, which is propagated from entity {1}, is not 
supported"



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 1420 (patched)


Are lines #1420 - #1424 necessary, since subsequent call to 
getPropagatedClassificationEdge() checks if the given classification is 
propagated from the given entity? If this block is not needed, please remove.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
Lines 1733 (patched)


Why copy to a set? Wouldn't 
existingClassifications.contains(suppliedClassificationName) do?


- Madhan Neethiraj


On Aug. 10, 2018, 2:36 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68287/
> ---
> 
> (Updated Aug. 10, 2018, 2:36 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and 
> Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-2807
> https://issues.apache.org/jira/browse/ATLAS-2807
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The current behavior - when we delete an entity f1, all tags associate to f1 
> which got propagated
> 
> 
> to downstream entities are removed – PII tag propagated to process1 and t1 is 
> removed
> 
> 
> Proposed Change
> 
> 
> Soft Delete:
> 
> 
> When entity f1 is deleted, retain the propagated classification edges, so the 
> downstream entities
> 
> 
> – process1 and t1 continue to have PII classification associated to them
> 
> 
> Hard Delete:
> 
> 
> In cases of hard delete, the source entity (f1) is deleted from atlas, but 
> its classification vertex
> 
> 
> (PII) continue to exist and will continue propagating to process1 and t1.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 7988d1d81 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 328b76734 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
> 3132379b9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> d328873f1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  c6be4061b 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  4a0924bbe 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
>  a95e6898e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java
>  41e65d4a3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  28a26ab4f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  f57ce99f7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  8f4faaff8 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  58005c458 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  8ceeedb01 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> e01cba92d 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java fd331faa0 
>   webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java 
> 78bd53ce7 
> 
> 
> Diff: https://reviews.apache.org/r/68287/diff/1/
> 
> 
> Testing
> ---
> 
> Validated basic cases using REST clirnt.
> 
> 
> PreCommit: In progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

[jira] [Commented] (ATLAS-2816) Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2

[Chengbing Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575843#comment-16575843
 ] 

Chengbing Liu commented on ATLAS-2816:
--

Thanks [~apoorvnaik] for review! Uploaded a new patch.

> Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2
> 
>
> Key: ATLAS-2816
> URL: https://issues.apache.org/jira/browse/ATLAS-2816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Chengbing Liu
>Assignee: Apoorv Naik
>Priority: Major
> Attachments: ATLAS-2816.01.patch, ATLAS-2816.02.patch
>
>
> We encountered a problem when using Hive bridge in production. One database 
> has 5000+ tables. Importing the first table costs only tens of milliseconds, 
> and then it becomes slower with more tables. In the end, it costs 1~2 seconds 
> to import one table.
> After investigation, we realized that it is not necessary for the 
> {{FullTextMapperV2}} to retrieve all the relationship of the database each 
> time a table is imported. The time complexity of importing a whole database 
> actually goes to O(n^2) (n is number of tables).
> We propose to add a parameter to the constructor of {{EntityGraphRetriever}}: 
> {{ignoreRelationship}}. When set to true, {{mapVertexToAtlasEntity}} will 
> skip the {{mapRelationshipAttributes}} call. Since {{FullTextMapperV2}} will 
> not use relationship attributes of the entity, this can save plenty of time 
> when importing entities with a large number of relations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ATLAS-2816) Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2

[Chengbing Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chengbing Liu updated ATLAS-2816:
-
Attachment: ATLAS-2816.02.patch

> Allow ignoring relationship in EntityGraphRetriever for FullTextMapperV2
> 
>
> Key: ATLAS-2816
> URL: https://issues.apache.org/jira/browse/ATLAS-2816
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Chengbing Liu
>Assignee: Apoorv Naik
>Priority: Major
> Attachments: ATLAS-2816.01.patch, ATLAS-2816.02.patch
>
>
> We encountered a problem when using Hive bridge in production. One database 
> has 5000+ tables. Importing the first table costs only tens of milliseconds, 
> and then it becomes slower with more tables. In the end, it costs 1~2 seconds 
> to import one table.
> After investigation, we realized that it is not necessary for the 
> {{FullTextMapperV2}} to retrieve all the relationship of the database each 
> time a table is imported. The time complexity of importing a whole database 
> actually goes to O(n^2) (n is number of tables).
> We propose to add a parameter to the constructor of {{EntityGraphRetriever}}: 
> {{ignoreRelationship}}. When set to true, {{mapVertexToAtlasEntity}} will 
> skip the {{mapRelationshipAttributes}} call. Since {{FullTextMapperV2}} will 
> not use relationship attributes of the entity, this can save plenty of time 
> when importing entities with a large number of relations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68287: ATLAS-2807: Re-evaluate classification propagation during entity delete

[David Radley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68287/#review207066
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
Line 56 (original)


I assume this is implementing the reporting text "Hard Delete:
In cases of hard delete, the source entity (f1) is deleted from atlas, but 
its classification vertex
(PII) continue to exist and will continue propagating to process1 and t1." 

I am not sure about this - as if a source entity is PII and it is deleted - 
I would think any downstream entities should not be PII.


- David Radley


On Aug. 10, 2018, 2:36 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68287/
> ---
> 
> (Updated Aug. 10, 2018, 2:36 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and 
> Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-2807
> https://issues.apache.org/jira/browse/ATLAS-2807
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The current behavior - when we delete an entity f1, all tags associate to f1 
> which got propagated
> 
> 
> to downstream entities are removed – PII tag propagated to process1 and t1 is 
> removed
> 
> 
> Proposed Change
> 
> 
> Soft Delete:
> 
> 
> When entity f1 is deleted, retain the propagated classification edges, so the 
> downstream entities
> 
> 
> – process1 and t1 continue to have PII classification associated to them
> 
> 
> Hard Delete:
> 
> 
> In cases of hard delete, the source entity (f1) is deleted from atlas, but 
> its classification vertex
> 
> 
> (PII) continue to exist and will continue propagating to process1 and t1.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 7988d1d81 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 328b76734 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
> 3132379b9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> d328873f1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  c6be4061b 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  4a0924bbe 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
>  a95e6898e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java
>  41e65d4a3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  28a26ab4f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  f57ce99f7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  8f4faaff8 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  58005c458 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  8ceeedb01 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> e01cba92d 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java fd331faa0 
>   webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java 
> 78bd53ce7 
> 
> 
> Diff: https://reviews.apache.org/r/68287/diff/1/
> 
> 
> Testing
> ---
> 
> Validated basic cases using REST clirnt.
> 
> 
> PreCommit: In progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Re: Review Request 68287: ATLAS-2807: Re-evaluate classification propagation during entity delete

[Graham Wallis]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68287/#review207065
---



Please look at the question I posted to the JIRA - I am not sure that you 
always want the specified behaviour.

- Graham Wallis


On Aug. 10, 2018, 2:36 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68287/
> ---
> 
> (Updated Aug. 10, 2018, 2:36 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and 
> Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-2807
> https://issues.apache.org/jira/browse/ATLAS-2807
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The current behavior - when we delete an entity f1, all tags associate to f1 
> which got propagated
> 
> 
> to downstream entities are removed – PII tag propagated to process1 and t1 is 
> removed
> 
> 
> Proposed Change
> 
> 
> Soft Delete:
> 
> 
> When entity f1 is deleted, retain the propagated classification edges, so the 
> downstream entities
> 
> 
> – process1 and t1 continue to have PII classification associated to them
> 
> 
> Hard Delete:
> 
> 
> In cases of hard delete, the source entity (f1) is deleted from atlas, but 
> its classification vertex
> 
> 
> (PII) continue to exist and will continue propagating to process1 and t1.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 7988d1d81 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 328b76734 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java 
> 3132379b9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> d328873f1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  c6be4061b 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  4a0924bbe 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
>  a95e6898e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java
>  41e65d4a3 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  28a26ab4f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  f57ce99f7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  8f4faaff8 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  58005c458 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  8ceeedb01 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> e01cba92d 
>   webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java fd331faa0 
>   webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java 
> 78bd53ce7 
> 
> 
> Diff: https://reviews.apache.org/r/68287/diff/1/
> 
> 
> Testing
> ---
> 
> Validated basic cases using REST clirnt.
> 
> 
> PreCommit: In progress
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>