date:20160414

[jira] [Commented] (ATLAS-622) Introduce soft delete

2016-04-14 Thread ATLAS QA (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242451#comment-15242451
 ] 

ATLAS QA commented on ATLAS-622:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12798892/ATLAS-622-v5.patch
  against master revision daf812a.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 20 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 113 javac 
compiler warnings (more than the master's current 109 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

+1 checkstyle.  The patch generated 0 code style errors.

{color:red}-1 findbugs{color}.  The patch appears to introduce 357 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   
com.thinkaurelius.titan.diskstorage.locking.LocalLockMediatorTest
./titan/target/surefire-reports/junitreports/TEST-com.thinkaurelius.titan.diskstorage.locking.LocalLockMediatorTest

Test results: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningswebapp.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningscommon.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningssqoop-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningshdfs-model.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningsstorm-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningshive-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningsrepository.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningstypesystem.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningsclient.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningsnotification.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/newPatchFindbugsWarningstitan.html
Javac warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/159//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-ATLAS-Build/159//console

This message is automatically generated.

> Introduce soft delete
> -
>
> Key: ATLAS-622
> URL: https://issues.apache.org/jira/browse/ATLAS-622
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Shwetha G S
>Assignee: Shwetha G S
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-622-v5.patch, ATLAS-622-v6.patch
>
>
> Currently, in entity delete API, the entity and its related 
> entities(composite entities) are deleted and there is no trace of it in the 
> system. Instead, change delete to mark the entities to be deleted with 
> state=DELETED



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/
---

(Updated April 15, 2016, 4:46 a.m.)


Review request for atlas.


Changes
---

Removed some un-necessary changes that were there


Bugs: ATLAS-622
https://issues.apache.org/jira/browse/ATLAS-622


Repository: atlas


Description
---

Added soft delete which is enabled by default. Hard delete can be configured 
using guice. Had to refactor entity deletes. 

Pending:
1. 
GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
 is disabled, need to debug
2. Have to write more tests for soft deletes
3. hive hook tests - have fixed some tests, 13/40 are still failing


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 f007a32 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
a0f1d40 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
dbba926 
  client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
  common/src/main/java/org/apache/atlas/ApplicationProperties.java ca72ffd 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
75d14f0 
  
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
 7f77feb 
  
repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java 
da2ad9a 
  repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
 b94ff5a 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 e7e8fb9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
895f9df 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
 1d682bb 
  
repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 7011fdf 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 71e9028 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java d07f89d 
  repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 ea93cbf 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
 975d530 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
 b5440c2 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
 3f71eb3 
  
repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
 8ca3132 
  
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 1156b67 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
  repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
cd1c424 
  repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
1a0fa20 
  server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
  typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
  typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
7857483 
  typesystem/src/main/java/org/apache/atlas/typesystem/Struct.java 70deab2 
  
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/DownCastStructInstance.java
 d3b9a33 
  typesystem/src/main/java/org/apache/atlas/typesystem/persistence/Id.java 
d742bb7 
  
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/ReferenceableInstance.java
 0fa4666 
  
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java
 16c3a24 
  typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
d8098ac 
  typesystem/src/test/resources/atlas-application.properties dbd6002 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java
 b2e9f91 
  
webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java 
23f6874

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS



> On April 14, 2016, 7:31 p.m., Suma Shivaprasad wrote:
> > repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java,
> >  line 122
> > 
> >
> > Didnt see the unique indexes being removed for all super types in 
> > DefaultMetadataService for qualifiedName and name attributes. Also in all 
> > the hook models for unique indexes. How is create after delete succeeding 
> > without this?

>From the modelling perspective, unique attributes still exist. But I removed 
>unique indexes in GraphBackedSearchIndexer so that titan doesn't enforce 
>uniqueness. Instead, they should be handled manually. In entity de-duping, 
>uniqueness is enforced on the first unique attribute of the class type. 
>However, unique attribute of struct/trait type or other unique attribute of 
>class type are not handled currently. I will fix that as part of another jira 
>- https://issues.apache.org/jira/browse/ATLAS-676


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128963
---


On April 15, 2016, 4:29 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 15, 2016, 4:29 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   common/src/main/java/org/apache/atlas/ApplicationProperties.java ca72ffd 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  7f77feb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
>

[jira] [Created] (ATLAS-676) Enforcing unique attribute values for struct/trait

2016-04-14 Thread Shwetha G S (JIRA)

Shwetha G S created ATLAS-676:
-

 Summary: Enforcing unique attribute values for struct/trait
 Key: ATLAS-676
 URL: https://issues.apache.org/jira/browse/ATLAS-676
 Project: Atlas
  Issue Type: Sub-task
Reporter: Shwetha G S
Assignee: Shwetha G S
 Fix For: 0.7-incubating


In ATLAS-622, titan doesn't handle unique attributes anymore and it needs to be 
handled manually. The jira takes care of enforcing uniqueness on first unique 
attribute of the class type as part of entity de-duping. But uniqueness on 
struct/trait attributes are not handled



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-622) Introduce soft delete

2016-04-14 Thread Shwetha G S (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S updated ATLAS-622:
--
Attachment: ATLAS-622-v5.patch

Complete patch for review

> Introduce soft delete
> -
>
> Key: ATLAS-622
> URL: https://issues.apache.org/jira/browse/ATLAS-622
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Shwetha G S
>Assignee: Shwetha G S
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-622-v5.patch
>
>
> Currently, in entity delete API, the entity and its related 
> entities(composite entities) are deleted and there is no trace of it in the 
> system. Instead, change delete to mark the entities to be deleted with 
> state=DELETED



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: apache-atlas-nightly #243

2016-04-14 Thread Apache Jenkins Server

See 

Changes:

[sumasai.shivaprasad] ATLAS-594 alter table rename doesnt work across databases 
( sumasai via

--
[...truncated 5928 lines...]
127.0.0.1 - - [14/Apr/2016:19:45:26 +] "GET 
/api/atlas/entities/22b3f77c-e739-4e76-af60-3fa0da074192/audit?count=100 
HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:27 +] "OPTIONS 
/api/atlas/types?user.name=jenkins HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:27 +] "POST /api/atlas/types HTTP/1.1" 201 
- "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:27 +] "OPTIONS 
/api/atlas/entities?user.name=jenkins HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:27 +] "POST /api/atlas/entities HTTP/1.1" 
201 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:27 +] "OPTIONS 
/api/atlas/types?user.name=jenkins HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:27 +] "PUT /api/atlas/types HTTP/1.1" 200 
- "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:27 +] "OPTIONS 
/api/atlas/entities/5f3eb76d-9987-4e62-b784-833f62251dd0?user.name=jenkins 
HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:27 +] "GET 
/api/atlas/entities/5f3eb76d-9987-4e62-b784-833f62251dd0 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "OPTIONS 
/api/atlas/entities?user.name=jenkins HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "POST /api/atlas/entities HTTP/1.1" 
400 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "OPTIONS 
/api/atlas/entities?user.name=jenkins HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "POST /api/atlas/entities HTTP/1.1" 
400 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "OPTIONS 
/api/atlas/entities?user.name=jenkins HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "POST /api/atlas/entities HTTP/1.1" 
201 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "OPTIONS 
/api/atlas/entities?type=hive_db=name=hZDvobR6wP=jenkins
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "GET 
/api/atlas/entities?type=hive_db=name=hZDvobR6wP HTTP/1.1" 200 - 
"-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "GET /api/atlas/entities?type=blah 
HTTP/1.1" 400 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "POST /api/atlas/types HTTP/1.1" 201 
- "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "GET 
/api/atlas/entities?type=testbL8hnyVfmM HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "GET /api/atlas/entities/blah 
HTTP/1.1" 404 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "OPTIONS 
/api/atlas/entities?user.name=jenkins HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "POST /api/atlas/entities HTTP/1.1" 
400 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "POST /api/atlas/entities HTTP/1.1" 
201 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "OPTIONS 
/api/atlas/types/%E5%92%98%EB%A5%AC%E7%B5%B8%EA%B9%9E%E5%A3%AB%EB%BC%B9%E7%94%A3%EF%83%B0%E4%A3%B6%E8%96%BB?user.name=jenkins
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "GET 
/api/atlas/types/%E5%92%98%EB%A5%AC%E7%B5%B8%EA%B9%9E%E5%A3%AB%EB%BC%B9%E7%94%A3%EF%83%B0%E4%A3%B6%E8%96%BB
 HTTP/1.1" 404 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:28 +] "POST /api/atlas/types HTTP/1.1" 201 
- "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:29 +] "OPTIONS 
/api/atlas/entities?user.name=jenkins HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:29 +] "POST /api/atlas/entities HTTP/1.1" 
201 - "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:29 +] "GET 
/api/atlas/entities/73be9525-0091-4a78-8254-8dc4561e693f HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
Tests run: 31, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.349 sec <<< 
FAILURE! - in org.apache.atlas.web.resources.EntityJerseyResourceIT
testEntityDeduping(org.apache.atlas.web.resources.EntityJerseyResourceIT)  Time 
elapsed: 1.271 sec  <<< FAILURE!
java.lang.Exception: Waiting timed out after 1000 msec
at 
org.apache.atlas.web.resources.EntityJerseyResourceIT.testEntityDeduping(EntityJerseyResourceIT.java:155)

Running org.apache.atlas.web.resources.TypesJerseyResourceIT
127.0.0.1 - - [14/Apr/2016:19:45:37 +] "POST /api/atlas/types HTTP/1.1" 201 
- "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:38 +] "POST /api/atlas/types HTTP/1.1" 201 
- "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:38 +] "POST /api/atlas/types HTTP/1.1" 201 
- "-" "Java/1.7.0_80"
127.0.0.1 - - [14/Apr/2016:19:45:38 +] "GET /api/atlas/types/database 
HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - -

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Suma Shivaprasad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128963
---




repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 (line 107)


Didnt see the unique indexes being removed for all super types in 
DefaultMetadataService for qualifiedName and name attributes. Also in all the 
hook models for unique indexes. How is create after delete succeeding without 
this?


- Suma Shivaprasad


On April 14, 2016, 10:29 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 14, 2016, 10:29 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   common/src/main/java/org/apache/atlas/ApplicationProperties.java ca72ffd 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  7f77feb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1156b67 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
> 1a0fa20 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
>

[jira] [Updated] (ATLAS-532) Change Data types of all timestamps(currently long)

2016-04-14 Thread Suma Shivaprasad (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suma Shivaprasad updated ATLAS-532:
---
Attachment: ATLAS-532.3.patch

Addressed review comments

> Change Data types of all timestamps(currently long)
> ---
>
> Key: ATLAS-532
> URL: https://issues.apache.org/jira/browse/ATLAS-532
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 0.6-incubating, 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-532.1.patch, ATLAS-532.2.patch, ATLAS-532.3.patch, 
> ATLAS-532.patch
>
>
> startTime, endTime, createTime to date



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-532) Change Data types of all timestamps(currently long)

2016-04-14 Thread Suma Shivaprasad (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suma Shivaprasad updated ATLAS-532:
---
Attachment: Screen Shot 2016-04-12 at 10.37.06 PM.png

> Change Data types of all timestamps(currently long)
> ---
>
> Key: ATLAS-532
> URL: https://issues.apache.org/jira/browse/ATLAS-532
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 0.6-incubating, 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-532.1.patch, ATLAS-532.2.patch, ATLAS-532.3.patch, 
> ATLAS-532.patch, Screen Shot 2016-04-12 at 10.37.06 PM.png
>
>
> startTime, endTime, createTime to date



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-532) Change Data types of all timestamps(currently long)

2016-04-14 Thread Suma Shivaprasad (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suma Shivaprasad updated ATLAS-532:
---
Attachment: Screen Shot 2016-04-08 at 2.40.24 PM.png

> Change Data types of all timestamps(currently long)
> ---
>
> Key: ATLAS-532
> URL: https://issues.apache.org/jira/browse/ATLAS-532
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 0.6-incubating, 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-532.1.patch, ATLAS-532.2.patch, ATLAS-532.3.patch, 
> ATLAS-532.patch
>
>
> startTime, endTime, createTime to date



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-532) Change Data types of all timestamps(currently long)

2016-04-14 Thread Suma Shivaprasad (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241736#comment-15241736
 ] 

Suma Shivaprasad commented on ATLAS-532:


Replies inline to [~yhemanth]'s review comments

We have defined constants for createTime etc in HiveMetaStoreBridge. It would 
be really nice if we could define these in HiveDataModelGenerator and use them 
both there and in other referring places. Similarly for the process related 
attributes.

   -> Fixed

In HiveHookIT.verifyTimestamps, there is an extraneous line: 
TypeSystem.getInstance().getDateFormat().parse(createTimeStr);. I don't think 
this is required, as it is being assigned to a variable before.

   -> Fixed

I saw that in Falcon and Storm we are still referring to timestamps as long 
datatypes. (Sqoop and HDFS seem fine). Ideally we would want to change those 
too. We could certainly take these as part of another JIRA. Could you please 
file that, if we take that path?

   --> Raised ATLAS-674 and ATLAS-675 for the same.

Would the API start serializing these differently now? If yes, any impact on 
existing apps including Åtlas's own UI that we need to call out? (Maybe mark 
these as incompatible changes and announce on the dev list etc)

   --> API serializes as String and UI is working fine with this. Attaching 
snapshot of tests

> Change Data types of all timestamps(currently long)
> ---
>
> Key: ATLAS-532
> URL: https://issues.apache.org/jira/browse/ATLAS-532
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 0.6-incubating, 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-532.1.patch, ATLAS-532.2.patch, ATLAS-532.3.patch, 
> ATLAS-532.patch
>
>
> startTime, endTime, createTime to date



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-532) Change Data types of all timestamps(currently long)

2016-04-14 Thread Suma Shivaprasad (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suma Shivaprasad updated ATLAS-532:
---
Attachment: (was: Screen Shot 2016-04-08 at 2.40.24 PM.png)

> Change Data types of all timestamps(currently long)
> ---
>
> Key: ATLAS-532
> URL: https://issues.apache.org/jira/browse/ATLAS-532
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 0.6-incubating, 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-532.1.patch, ATLAS-532.2.patch, ATLAS-532.3.patch, 
> ATLAS-532.patch
>
>
> startTime, endTime, createTime to date



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-675) Storm Hook should use timetsamps as Date type instead of Long

2016-04-14 Thread Suma Shivaprasad (JIRA)

Suma Shivaprasad created ATLAS-675:
--

 Summary: Storm Hook should use timetsamps as Date type instead of 
Long
 Key: ATLAS-675
 URL: https://issues.apache.org/jira/browse/ATLAS-675
 Project: Atlas
  Issue Type: Bug
Reporter: Suma Shivaprasad


Storm topology startTime and endTime should be of type Date



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-674) Falcon Hook should use timestamps instead of long

2016-04-14 Thread Suma Shivaprasad (JIRA)

Suma Shivaprasad created ATLAS-674:
--

 Summary: Falcon Hook should use timestamps instead of long
 Key: ATLAS-674
 URL: https://issues.apache.org/jira/browse/ATLAS-674
 Project: Atlas
  Issue Type: Bug
Reporter: Suma Shivaprasad


For eg: TIMESTAMP attribute is of type LONG. should be of type Date.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New Committer: Hemanth Yamijala

2016-04-14 Thread Suma Shivaprasad

Congrats Hemanth!

On Thu, Apr 14, 2016 at 7:05 AM, Erik Bergenholtz <
ebergenho...@hortonworks.com> wrote:

> That’s great news! Congrats & well deserved.
>
> Erik
>
> > On Apr 14, 2016, at 1:15 AM, Shwetha Shivalingamurthy <
> sshivalingamur...@hortonworks.com> wrote:
> >
> > Hi All,
> >
> > The Apache Atlas PPMC has asked Hemanth Yamijala to become a committer
> and we are pleased to announce that he has accepted.
> >
> > Regards,
> > Shwetha
> >
>
>

[jira] [Commented] (ATLAS-673) Install Setup: Kafka & SOLR

2016-04-14 Thread Erik Bergenholtz (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241354#comment-15241354
 ] 

Erik Bergenholtz commented on ATLAS-673:


[~yhemanth] - can you elaborate on the setup requirement to make sure that it 
is captured properly. The above does not captures the details.

> Install Setup: Kafka & SOLR
> ---
>
> Key: ATLAS-673
> URL: https://issues.apache.org/jira/browse/ATLAS-673
> Project: Atlas
>  Issue Type: Task
>Affects Versions: 0.7-incubating
>Reporter: Erik Bergenholtz
>Assignee: Hemanth Yamijala
>Priority: Critical
> Fix For: 0.7-incubating
>
>
> As a part of Atlas install, Kafka topics with appropriate settings as well as 
> SOLR indexes need to be setup properly. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-673) Install Setup: Kafka & SOLR

2016-04-14 Thread Erik Bergenholtz (JIRA)

Erik Bergenholtz created ATLAS-673:
--

 Summary: Install Setup: Kafka & SOLR
 Key: ATLAS-673
 URL: https://issues.apache.org/jira/browse/ATLAS-673
 Project: Atlas
  Issue Type: Task
Affects Versions: 0.7-incubating
Reporter: Erik Bergenholtz
Assignee: Hemanth Yamijala
Priority: Critical
 Fix For: 0.7-incubating


As a part of Atlas install, Kafka topics with appropriate settings as well as 
SOLR indexes need to be setup properly. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at the same time causes exceptions in HA mode.

2016-04-14 Thread Hemanth Yamijala


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/
---

(Updated April 14, 2016, 3:24 p.m.)


Review request for atlas.


Changes
---

Added the ability to run the setup steps as part of Atlas server start for 
those who don't want to rely on an additional setup step. This is enabled by a 
configuration (disabled by default) if required.


Bugs: ATLAS-628
https://issues.apache.org/jira/browse/ATLAS-628


Repository: atlas


Description
---

Implements capability in atlas to run a set of registered setup steps. 
Currently the only step is setting up the backend graph repository. The intent 
is to extend this later for setting up Kafka at least and maybe other things 
like Solr.


Diffs (updated)
-

  distro/src/bin/atlas_start.py 7db74c0 
  distro/src/conf/atlas-application.properties 4131240 
  docs/src/site/twiki/Configuration.twiki 023f5a0 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java
 PRE-CREATION 
  server-api/src/main/java/org/apache/atlas/setup/SetupException.java 
PRE-CREATION 
  server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d 
  webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e 
  webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java 
PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION 
  webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/45929/diff/


Testing
---

Existing ITs, UTs pass.

* Manually verified setting up with both embedded HBase setup and external 
HBase setup.
* Also ran atlas HA instances at exactly same time and verified the noted 
exception does not happen.


Thanks,

Hemanth Yamijala

[jira] [Assigned] (ATLAS-559) Atlas should allow JSONs to be submitted that don't have optional attributes.

2016-04-14 Thread Neeru Gupta (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neeru Gupta reassigned ATLAS-559:
-

Assignee: Neeru Gupta

> Atlas should allow JSONs to be submitted that don't have optional attributes.
> -
>
> Key: ATLAS-559
> URL: https://issues.apache.org/jira/browse/ATLAS-559
> Project: Atlas
>  Issue Type: Bug
>Reporter: Hemanth Yamijala
>Assignee: Neeru Gupta
> Fix For: 0.7-incubating
>
>
> Currently, attributes which are in principle optional should still be 
> submitted using a null value if one is using the REST API directly. The Java 
> client libraries handle this case for the user, but there are non Java folks 
> using Atlas as well. For them, any optional addition to the types implies 
> they will need to change their code which is super inconvenient. Hence, 
> handling optional attributes is required.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread David Kantor



> On April 12, 2016, 10:20 p.m., David Kantor wrote:
> > repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java, 
> > line 90
> > 
> >
> > Configuring hard deletes at the system level should be done via the 
> > Atlas configuration properties (i.e. atlas-application.properties) or 
> > something that can be configured by Atlas consumers - not a compile-time 
> > mechanism like Guice.  Also, for our use cases, it would be strongly 
> > preferable to enable hard delete at the API and/or type level and not be 
> > limited to a system-wide configuration.
> 
> Shwetha GS wrote:
> I will make the delete configurable using application properties. But 
> hard delete/soft delete at API/type level has to be done in another jira

Thanks for addressing this comment. Implementing API/type level configuration 
in another JIRA makes sense, as it is arguably beyond the scope of this initial 
task.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128551
---


On April 14, 2016, 10:29 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 14, 2016, 10:29 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   common/src/main/java/org/apache/atlas/ApplicationProperties.java ca72ffd 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  7f77feb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
>

[jira] [Resolved] (ATLAS-436) Add LIMIT to search DSL

2016-04-14 Thread Neeru Gupta (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neeru Gupta resolved ATLAS-436.
---
Resolution: Duplicate

Duplicate of ATLAS-435

> Add LIMIT to search DSL
> ---
>
> Key: ATLAS-436
> URL: https://issues.apache.org/jira/browse/ATLAS-436
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Kantor
>Assignee: Neeru Gupta
>
> Add LIMIT to Atlas search DSL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-435) Add ORDER BY to search DSL

2016-04-14 Thread Neeru Gupta (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241224#comment-15241224
 ] 

Neeru Gupta commented on ATLAS-435:
---

[~suma.shivaprasad] Please review the changes for ATLAS-435 and ATLAS-436 at 
https://reviews.apache.org/r/45499/ . [~dkantor] has mentioned that you are 
familiar with DSL parsing code. Thanks Neeru

> Add ORDER BY to search DSL 
> ---
>
> Key: ATLAS-435
> URL: https://issues.apache.org/jira/browse/ATLAS-435
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Kantor
>Assignee: Neeru Gupta
>
> Add support for ORDER BY to Atlas search DSL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 45499: ATLAS-436, ATLAS-435 order by and limit clause for DSL queries

2016-04-14 Thread Neeru Gupta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45499/
---

(Updated April 14, 2016, 2:11 p.m.)


Review request for atlas, David Kantor and Jeff Hagelberg.


Bugs: ATLAS-435 and ATLAS-436
https://issues.apache.org/jira/browse/ATLAS-435
https://issues.apache.org/jira/browse/ATLAS-436


Repository: atlas


Description
---

ATLAS-435, 436 order by and limit clause in DSL


Diffs (updated)
-

  repository/src/main/scala/org/apache/atlas/query/Expressions.scala 
a5dfa9f409c0d810be7449413f5c2d53d5103dce 
  repository/src/main/scala/org/apache/atlas/query/GremlinEvaluator.scala 
edb190db599d43eb98eca13c7907c73e7d77ae34 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 
f1590a884eda007182dad7541a2bc8eb97d68fbc 
  repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 
b6bbbd31a91dd2d0ff3e9082abafff3539765a01 
  repository/src/main/scala/org/apache/atlas/query/Resolver.scala 
c7e1e81a555d57be1d7f63cb84a1c8f550ac37ab 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 ea93cbf0e6071a961aa5d030f0340358e79e446b 

Diff: https://reviews.apache.org/r/45499/diff/


Testing
---

Manual and unit testing done for various DSL queries. Have added test cases in 
GraphBackedDiscoveryServiceTest for orderby and limit clause for various 
queries.

Order by clause is limited to specifying one criterian only. Comparison is case 
insensitive.


Thanks,

Neeru Gupta

[jira] [Commented] (ATLAS-672) UI: Make dashboard v2 the default UI implementation

2016-04-14 Thread JIRA


[ 
https://issues.apache.org/jira/browse/ATLAS-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241198#comment-15241198
 ] 

Jean-Baptiste Onofré commented on ATLAS-672:


By the way, I'm testing dashboard v2 now.

> UI: Make dashboard v2 the default UI implementation
> ---
>
> Key: ATLAS-672
> URL: https://issues.apache.org/jira/browse/ATLAS-672
> Project: Atlas
>  Issue Type: Task
>Affects Versions: 0.7-incubating
>Reporter: Erik Bergenholtz
>Assignee: Erik Bergenholtz
> Fix For: 0.7-incubating
>
>
> Now that dasboardv2 has been a part of the code base for some time, I propose 
> that we move to using this implementation in favor of the older dashboard 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-672) UI: Make dashboard v2 the default UI implementation

2016-04-14 Thread JIRA


[ 
https://issues.apache.org/jira/browse/ATLAS-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241191#comment-15241191
 ] 

Jean-Baptiste Onofré commented on ATLAS-672:


+1

> UI: Make dashboard v2 the default UI implementation
> ---
>
> Key: ATLAS-672
> URL: https://issues.apache.org/jira/browse/ATLAS-672
> Project: Atlas
>  Issue Type: Task
>Affects Versions: 0.7-incubating
>Reporter: Erik Bergenholtz
>Assignee: Erik Bergenholtz
> Fix For: 0.7-incubating
>
>
> Now that dasboardv2 has been a part of the code base for some time, I propose 
> that we move to using this implementation in favor of the older dashboard 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-672) UI: Make dashboard v2 the default UI implementation

2016-04-14 Thread Erik Bergenholtz (JIRA)

Erik Bergenholtz created ATLAS-672:
--

 Summary: UI: Make dashboard v2 the default UI implementation
 Key: ATLAS-672
 URL: https://issues.apache.org/jira/browse/ATLAS-672
 Project: Atlas
  Issue Type: Task
Affects Versions: 0.7-incubating
Reporter: Erik Bergenholtz
Assignee: Erik Bergenholtz
 Fix For: 0.7-incubating


Now that dasboardv2 has been a part of the code base for some time, I propose 
that we move to using this implementation in favor of the older dashboard 
implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 45499: ATLAS-436, ATLAS-435 order by and limit clause for DSL queries

2016-04-14 Thread Jeff Hagelberg



> On April 7, 2016, 7:55 p.m., Jeff Hagelberg wrote:
> > repository/src/main/scala/org/apache/atlas/query/GremlinEvaluator.scala, 
> > line 99
> > 
> >
> > I would suggest adding an intermediate variable here for the output of 
> > the if expression to improve readability.  Generally, the longer an 
> > expression is, the harder it is to understand.
> 
> Neeru Gupta wrote:
> May be I am missing something, but there is already a output variable: 
> val selExpr. This variable gets initialized based on the parent expression.

This is a very minor comment.  The thing that stood out was the period at the 
end of line 100.  The expression is effectively val selExpr = [result of three 
part if statement].asInstanceOf[Expressions.SelectExpression]

I was thinking this could be split into two or more statements, something like 
this:

val selObj = [three part if statement]
val selExpr = selObj.asInstanceOf[Expressions.SelectExpression]


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45499/#review127643
---


On April 14, 2016, 2:15 a.m., Neeru Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45499/
> ---
> 
> (Updated April 14, 2016, 2:15 a.m.)
> 
> 
> Review request for atlas, David Kantor and Jeff Hagelberg.
> 
> 
> Bugs: ATLAS-435 and ATLAS-436
> https://issues.apache.org/jira/browse/ATLAS-435
> https://issues.apache.org/jira/browse/ATLAS-436
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-435, 436 order by and limit clause in DSL
> 
> 
> Diffs
> -
> 
>   repository/src/main/scala/org/apache/atlas/query/Expressions.scala 
> a5dfa9f409c0d810be7449413f5c2d53d5103dce 
>   repository/src/main/scala/org/apache/atlas/query/GremlinEvaluator.scala 
> edb190db599d43eb98eca13c7907c73e7d77ae34 
>   repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 
> f1590a884eda007182dad7541a2bc8eb97d68fbc 
>   repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 
> b6bbbd31a91dd2d0ff3e9082abafff3539765a01 
>   repository/src/main/scala/org/apache/atlas/query/Resolver.scala 
> c7e1e81a555d57be1d7f63cb84a1c8f550ac37ab 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf0e6071a961aa5d030f0340358e79e446b 
> 
> Diff: https://reviews.apache.org/r/45499/diff/
> 
> 
> Testing
> ---
> 
> Manual and unit testing done for various DSL queries. Have added test cases 
> in GraphBackedDiscoveryServiceTest for orderby and limit clause for various 
> queries.
> 
> Order by clause is limited to specifying one criterian only. Comparison is 
> case insensitive.
> 
> 
> Thanks,
> 
> Neeru Gupta
> 
>

Review Request 46197: ATLAS-658 : Improve Lineage with Backbone porting

2016-04-14 Thread keval bhatt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46197/
---

Review request for atlas, Erik Bergenholtz, Gautam Borad, Mehul Parikh, Nixon 
Rodrigues, and Hemanth Yamijala.


Bugs: ATLAS-658
https://issues.apache.org/jira/browse/ATLAS-658


Repository: atlas


Description
---

Improved display of vertices and edges in lineage graph that gets generated.


Diffs
-

  dashboardv2/public/css/override.css 4c452cc 
  dashboardv2/public/css/style.css d1d23cc 
  dashboardv2/public/js/collection/BaseCollection.js 2018019 
  dashboardv2/public/js/collection/VCommonList.js d9e1f52 
  dashboardv2/public/js/collection/VSchemaList.js 6a5c9d6 
  dashboardv2/public/js/collection/VTagList.js 9fa48bf 
  dashboardv2/public/js/modules/Helpers.js 4deb19b 
  dashboardv2/public/js/templates/common/TableLayout_tmpl.html 4264035 
  dashboardv2/public/js/templates/common/modal.html 0f64f5a 
  dashboardv2/public/js/templates/detail_page/DetailPageLayoutView_tmpl.html 
3a8eb44 
  dashboardv2/public/js/templates/graph/LineageLayoutView_tmpl.html 0c32a48 
  dashboardv2/public/js/templates/schema/SchemaTableLayoutView_tmpl.html 
cbc0e8a 
  dashboardv2/public/js/templates/tag/addTagModalView_tmpl.html acae4e1 
  dashboardv2/public/js/utils/Overrides.js 3b598e3 
  dashboardv2/public/js/utils/Utils.js 7b2db79 
  dashboardv2/public/js/views/asset/AssetPageLayoutView.js c86fef4 
  dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js 87adec0 
  dashboardv2/public/js/views/graph/LineageLayoutView.js 973d091 
  dashboardv2/public/js/views/schema/SchemaLayoutView.js 35487a5 
  dashboardv2/public/js/views/site/Header.js 500bd91 
  dashboardv2/public/js/views/tag/TagLayoutView.js 42791b9 
  dashboardv2/public/js/views/tag/addTagModalView.js d663ae2 
  dashboardv2/public/js/views/tag/createTagsLayoutView.js 5d93eee 

Diff: https://reviews.apache.org/r/46197/diff/


Testing
---

1) mvn tests
2) Loaded the new UI in local sandbox and verified Lineage graph generation 
with proper vcertices and edges shown based on input and output of API calls.


Thanks,

keval bhatt

[jira] [Updated] (ATLAS-658) Improve Lineage with Backbone porting

2016-04-14 Thread Keval Bhatt (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt updated ATLAS-658:
--
Attachment: ATLAS-658.patch

> Improve Lineage with Backbone porting 
> --
>
> Key: ATLAS-658
> URL: https://issues.apache.org/jira/browse/ATLAS-658
> Project: Atlas
>  Issue Type: Bug
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-658.patch
>
>
> Improved display of vertices and edges in lineage graph that gets generated. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-671) Include binary dist in release

2016-04-14 Thread JIRA

Jean-Baptiste Onofré created ATLAS-671:
--

 Summary: Include binary dist in release
 Key: ATLAS-671
 URL: https://issues.apache.org/jira/browse/ATLAS-671
 Project: Atlas
  Issue Type: Task
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré
 Fix For: 0.7-incubating


The 0.6-incubating release is not available on Maven Central (whereas it is on 
{{dist.apache.org}}).
In addition of the source distribution, it would be helpful for users to be 
able to find binary distribution on Central.

I propose to enable dist profile on the maven-release-plugin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-670) Entity update/delete operations shouldn't be supported on deleted entities

2016-04-14 Thread Shwetha G S (JIRA)

Shwetha G S created ATLAS-670:
-

 Summary: Entity update/delete operations shouldn't be supported on 
deleted entities
 Key: ATLAS-670
 URL: https://issues.apache.org/jira/browse/ATLAS-670
 Project: Atlas
  Issue Type: Sub-task
Reporter: Shwetha G S
Assignee: Shwetha G S






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-594) alter table rename doesnt work across databases

2016-04-14 Thread Shwetha G S (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241011#comment-15241011
 ] 

Shwetha G S commented on ATLAS-594:
---

+1

> alter table rename doesnt work across databases
> ---
>
> Key: ATLAS-594
> URL: https://issues.apache.org/jira/browse/ATLAS-594
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 0.6-incubating, 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-594.patch
>
>
> Alter table rename across dbs doesnt work – alter table db1.tb1 rename to 
> db2.tb2 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-543) Entity Instance requests should not require ID element for new Entities

2016-04-14 Thread Shwetha G S (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240998#comment-15240998
 ] 

Shwetha G S commented on ATLAS-543:
---

+1

> Entity Instance requests should not require ID element for new Entities
> ---
>
> Key: ATLAS-543
> URL: https://issues.apache.org/jira/browse/ATLAS-543
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.6-incubating
> Environment: Ubuntu 14, OpenJDK 64-Bit 1.7.0_95
>Reporter: Joseph Niemiec
>Assignee: Harish Jaiprakash
>Priority: Minor
> Attachments: ATLAS-543.patch
>
>
> When utilizing the REST API to create an 'new' Entity Instance of a given 
> Type all ID elements for the class and structs are required, requests without 
> these elements will fail despite the fact a random GUID will be assigned at 
> instance time. 
> #
> Example 1 (Good Entity Posts correctly) 
> #
> {
>   "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>   "id": {
>   "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>   "id": "-984848",
>   "version": 0,
>   "typeName": "HDFS_RESOURCE"
>   },
>   "typeName": "HDFS_RESOURCE",
>   "values": {
>   "name": "Cluser_A_DevFolder_A",
>   "description": "Fully Public Dev Folder",
>   "resource": {
>   "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>   "id": {
>   "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>   "id": "-2630837415522",
>   "version": 0,
>   "typeName": "HDFS_OBJECT"
>   },
>   "typeName": "HDFS_OBJECT",
>   "values": {
>   "uri": "/user/dev/a",
>   "isDir" : true
>   },
>   "traitNames": [],
>   "traits": {}
>   }
>   },
>   "traitNames": ["Public"],
>   "traits": {
>   "Public": {
>   "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
>   "typeName": "Public",
>   "values": { }
>   }
>   }
> }
> 
> Example #2 Bad Entity that fails.
> 
> {
>   "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>   "typeName": "HDFS_RESOURCE",
>   "values": {
>   "name": "Cluser_A_DevFolder_A",
>   "description": "Fully Public Dev Folder",
>   "resource": {
>   "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>   "typeName": "HDFS_OBJECT",
>   "values": {
>   "uri": "/user/dev/a",
>   "isDir" : true
>   },
>   "traitNames": [],
>   "traits": {}
>   }
>   },
>   "traitNames": ["Public"],
>   "traits": {
>   "Public": {
>   "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
>   "typeName": "Public",
>   "values": { }
>   }
>   }
> }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-494) UI Authentication

2016-04-14 Thread Shwetha G S (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S updated ATLAS-494:
--
Summary: UI Authentication   (was: Authentication )

> UI Authentication 
> --
>
> Key: ATLAS-494
> URL: https://issues.apache.org/jira/browse/ATLAS-494
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 0.7-incubating
>Reporter: Erik Bergenholtz
>Assignee: Nixon Rodrigues
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-494.1.patch, ATLAS-494.2.patch, ATLAS-494.3.patch, 
> ATLAS-494.4.patch, ATLAS-494.patch
>
>
> Atlas needs an authentication mechanism besides kerberos.
> Requirements
> - Provide ability to authenticate against local file
> - Provide Authentication that supports LDAP
> - For consistency, configuration should use exactly the same configuration 
> (properties) as Ranger & Knox



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-664) UI: Add Support for Versioning / History

2016-04-14 Thread Keval Bhatt (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt updated ATLAS-664:
--
Assignee: Kalyani Kashikar  (was: Keval Bhatt)

> UI: Add Support for Versioning / History
> 
>
> Key: ATLAS-664
> URL: https://issues.apache.org/jira/browse/ATLAS-664
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 0.7-incubating
>Reporter: Erik Bergenholtz
>Assignee: Kalyani Kashikar
> Fix For: 0.7-incubating
>
>
> ATLAS-540 adds REST API support for fetching information about changes to 
> entities stored in Atlas (this is being referred to as versioning). For each 
> detailed page in Atlas, we need to expose the information provided by the 
> REST API in tabular format.
> It makes sense to create a separate tab (next to "tags") on the entity 
> detailed page where this information is displayed.
> The API is straightforward and can be accessed by {guid}/audit
> For more information see:
> https://github.com/apache/incubator-atlas/blob/master/webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java#L612



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-616) Zookeeper throws exceptions when trying to fire DSL queries at Atlas at large scale.

2016-04-14 Thread Hemanth Yamijala (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hemanth Yamijala updated ATLAS-616:
---
Attachment: heap.png

An update:

As described above, all indications to cause of the problem were pointing 
towards the weak references that were holding on the GremlinGroovy script 
bindings. From what I could see in the code, there are no knobs to adjust / 
tune this value in the version of the library we are using.

As a next step, I tried to see whether GC settings could be tuned to accomplish 
this, and ran across this link: http://stackoverflow.com/a/604395 which pointed 
to a GC config {{-XX:SoftRefLRUPolicyMSPerMB=}}. Likewise, the Sun JDK 
documentation 
(http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html) says: 

bq. -XX:SoftRefLRUPolicyMSPerMB=0 This flag enables aggressive processing of 
software references. Use this flag if the software reference count has an 
impact on the Java HotSpot VM garbage collector.

Given the above hints, I ran a test with this setting, set to 0 and also to 
100. In both cases, the GC performance dramatically improved and I was able to 
increase the number of tests to get linear performance. [~ssainath] helped me 
to run these tests in a server environment (still with JDK 7) and got similar 
results. The attached graph is from a server environment running a total of 
3600 queries. We even tested up to 7200 queries. Each run scaled linearly with 
time, and the logs had no concurrency issues etc. The GC patterns are stable as 
can be seen above.

We are going to test on OpenJDK 8 as well to see what the impact is, and if 
things go fine, I can put up a patch that just suggests the settings to enable 
on the server for such loads.

For reference, the GC settings I use are:
{code}
export ATLAS_OPTS="-server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:MaxNewSize=3072m 
-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC 
-XX:+CMSParallelRemarkEnabled -XX:MaxPermSize=512m 
-Djava.net.preferIPv4Stack=true  -Xmx10240m -Xms10240m 
-XX:+PrintTenuringDistribution  -XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=dumps/atlas_server.hprof -XX:PermSize=100M 
-Xloggc:logs/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation 
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails 
-XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -Dlog4j.configuration=atlas-log4j.xml"
{code}

In addition to this effort, I also plan to write on the Tinkerpop mailing list 
to see if they have any suggestions for tuning this / fixing this in code.

> Zookeeper throws exceptions when trying to fire DSL queries at Atlas at large 
> scale. 
> -
>
> Key: ATLAS-616
> URL: https://issues.apache.org/jira/browse/ATLAS-616
> Project: Atlas
>  Issue Type: Bug
> Environment: Atlas with External kafka / HBase / Solr
> The test is run on cluster setup.
> Machine 1 - Atlas , Solr
> Machine 2 - Kafka , HBase
> Machine 3 - Hive , client
>Reporter: Sharmadha Sainath
>Assignee: Hemanth Yamijala
> Attachments: baseline-1000-3600-10g-heap.png, heap.png, 
> no-dsl-1000-14400-10g-heap.png, zk-exception-stacktrace.rtf
>
>
> The test plan is to simulate 'n' number of users fire 'm' number of queries 
> at Atlas simultaneously. This is accomplished with the help of Apache Jmeter.
> Atlas is populated with 10,000 tables. 
> • 6000 small sized tables (10 columns)
> • 3000 medium sized tables (50 columns)
> • 1000 large sized tables (100 columns)
>  The test plan consists of 30 users firing a set of 3 queries continuously 
> for 20 times in a loop. Added -Xmx10240m -XX:MaxPermSize=512m to ATLAS_OPTS . 
> Zookeeper throws exceptions when the test plan is run and Jmeter starts 
> firing queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS



> On April 13, 2016, 10:36 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java,
> >  line 184
> > 
> >
> > There could be a way to get the sorted edge list by {state, 
> > modification time} from Titan itself by adding the edge index.

Created https://issues.apache.org/jira/browse/ATLAS-669. Lets address this 
separately


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128775
---


On April 14, 2016, 10:29 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 14, 2016, 10:29 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   common/src/main/java/org/apache/atlas/ApplicationProperties.java ca72ffd 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  7f77feb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1156b67 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
> 1a0fa20 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>

[jira] [Created] (ATLAS-669) GraphHelper.getEdgeForLabel() can use edge index

2016-04-14 Thread Shwetha G S (JIRA)

Shwetha G S created ATLAS-669:
-

 Summary: GraphHelper.getEdgeForLabel() can use edge index
 Key: ATLAS-669
 URL: https://issues.apache.org/jira/browse/ATLAS-669
 Project: Atlas
  Issue Type: Bug
Reporter: Shwetha G S


GraphHelper.getEdgeForLabel()  currently goes through all edges and gets the 
latest deleted edge. This can be optimised and use edge index and ordering



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/
---

(Updated April 14, 2016, 10:29 a.m.)


Review request for atlas.


Changes
---

Complete patch with the review comments addressed and the tests pass.


Bugs: ATLAS-622
https://issues.apache.org/jira/browse/ATLAS-622


Repository: atlas


Description
---

Added soft delete which is enabled by default. Hard delete can be configured 
using guice. Had to refactor entity deletes. 

Pending:
1. 
GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
 is disabled, need to debug
2. Have to write more tests for soft deletes
3. hive hook tests - have fixed some tests, 13/40 are still failing


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 f007a32 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
a0f1d40 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
dbba926 
  client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
  common/src/main/java/org/apache/atlas/ApplicationProperties.java ca72ffd 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
75d14f0 
  
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
 7f77feb 
  
repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java 
da2ad9a 
  repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
 b94ff5a 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 e7e8fb9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
895f9df 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
 1d682bb 
  
repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 7011fdf 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 71e9028 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java d07f89d 
  repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 ea93cbf 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
 975d530 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
 b5440c2 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
 3f71eb3 
  
repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
 8ca3132 
  
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 1156b67 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
  repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
cd1c424 
  repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
1a0fa20 
  server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
  typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
  typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
7857483 
  typesystem/src/main/java/org/apache/atlas/typesystem/Struct.java 70deab2 
  
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/DownCastStructInstance.java
 d3b9a33 
  typesystem/src/main/java/org/apache/atlas/typesystem/persistence/Id.java 
d742bb7 
  
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/ReferenceableInstance.java
 0fa4666 
  
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java
 16c3a24 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
55ec91f 
  typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
d8098ac 
  
typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java
 eb1a15a

[jira] [Commented] (ATLAS-668) GraphHelper.addEdge() should add an edge only if there is no existing edge

2016-04-14 Thread Hemanth Yamijala (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240905#comment-15240905
 ] 

Hemanth Yamijala commented on ATLAS-668:


Yes, of course.. sorry I missed that.

> GraphHelper.addEdge() should add an edge only if there is no existing edge
> --
>
> Key: ATLAS-668
> URL: https://issues.apache.org/jira/browse/ATLAS-668
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>
> Currently, addEdge adds edge always and because of any code issues, we might 
> end with duplicate edges. addEdge should add a check before adding edge



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-668) GraphHelper.addEdge() should add an edge only if there is no existing edge

2016-04-14 Thread Shwetha G S (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240900#comment-15240900
 ] 

Shwetha G S commented on ATLAS-668:
---

GraphBackedTypeStore is just for type store. Entity store is in 
GraphBackedMetadataRepository and uses GraphHelper. We should change 
GraphBackedTypeStore to use GraphHelper

> GraphHelper.addEdge() should add an edge only if there is no existing edge
> --
>
> Key: ATLAS-668
> URL: https://issues.apache.org/jira/browse/ATLAS-668
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>
> Currently, addEdge adds edge always and because of any code issues, we might 
> end with duplicate edges. addEdge should add a check before adding edge



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (ATLAS-664) UI: Add Support for Versioning / History

2016-04-14 Thread Keval Bhatt (JIRA)


 [ 
https://issues.apache.org/jira/browse/ATLAS-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt reassigned ATLAS-664:
-

Assignee: Keval Bhatt  (was: Nixon Rodrigues)

> UI: Add Support for Versioning / History
> 
>
> Key: ATLAS-664
> URL: https://issues.apache.org/jira/browse/ATLAS-664
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 0.7-incubating
>Reporter: Erik Bergenholtz
>Assignee: Keval Bhatt
> Fix For: 0.7-incubating
>
>
> ATLAS-540 adds REST API support for fetching information about changes to 
> entities stored in Atlas (this is being referred to as versioning). For each 
> detailed page in Atlas, we need to expose the information provided by the 
> REST API in tabular format.
> It makes sense to create a separate tab (next to "tags") on the entity 
> detailed page where this information is displayed.
> The API is straightforward and can be accessed by {guid}/audit
> For more information see:
> https://github.com/apache/incubator-atlas/blob/master/webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java#L612



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS



> On April 12, 2016, 6:57 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java,
> >  line 244
> > 
> >
> > Please enable this check. Raising this to track so that we dont miss

Created https://issues.apache.org/jira/browse/ATLAS-667


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128505
---


On April 13, 2016, 4:12 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 13, 2016, 4:12 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1156b67 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
> 1a0fa20 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 7857483 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Struct.java 70deab2 
>   
>

[jira] [Commented] (ATLAS-668) GraphHelper.addEdge() should add an edge only if there is no existing edge

2016-04-14 Thread Hemanth Yamijala (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240847#comment-15240847
 ] 

Hemanth Yamijala commented on ATLAS-668:


Umm. As a fix for ATLAS-474, a check was added to 
{{GraphBackedTypeStore.addEdge}} to avoid this condition. So all calls to Titan 
don't go through the GraphBackedTypeStore? Should they?

> GraphHelper.addEdge() should add an edge only if there is no existing edge
> --
>
> Key: ATLAS-668
> URL: https://issues.apache.org/jira/browse/ATLAS-668
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>
> Currently, addEdge adds edge always and because of any code issues, we might 
> end with duplicate edges. addEdge should add a check before adding edge



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS



> On April 12, 2016, 7:51 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java,
> >  line 111
> > 
> >
> > Can you pls explain how deletion of edges works with soft deletes? I 
> > noticed that in case of isComposite alone we are deleting the edges 
> > forcefully(force flag) where in case of struct, array, map - this is not 
> > the case. I had assumed that in case of soft deletes  we will not delete 
> > the edges even in teh case of composites

Introduced state on edges and edges are marked as deleted.  There is also 
create time and last modified time on edges. We can actually get point in time 
entity state using this and entity audit

For composite array attributes, only the active references are returned. For 
example, for hive_table, only active columns are returned
For non-composite array attributes, even the deleted references are returned. 
For example, for hive_process, deleted input and output tables are returned


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128517
---


On April 13, 2016, 4:12 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 13, 2016, 4:12 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1156b67 
>

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS



> On April 12, 2016, 6:23 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java,
> >  line 35
> > 
> >
> > why is this removed?

Its used even for trait edges which don't have this prefix


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128447
---


On April 13, 2016, 4:12 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 13, 2016, 4:12 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1156b67 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
> 1a0fa20 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 7857483 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Struct.java 70deab2 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/DownCastStructInstance.java
>  d3b9a33 
>

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS



> On April 11, 2016, 10:11 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java, 
> > line 90
> > 
> >
> > How are we planing to make this configurable? This is configurable at 
> > system or type level?

This will be at system level for now. Type level customisation can be added 
later

Currently, its configured through guice. Will make it configurable through 
application properties


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128270
---


On April 13, 2016, 4:12 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 13, 2016, 4:12 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1156b67 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
> 1a0fa20 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 7857483 
>

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS



> On April 12, 2016, 10:20 p.m., David Kantor wrote:
> > repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java, 
> > line 90
> > 
> >
> > Configuring hard deletes at the system level should be done via the 
> > Atlas configuration properties (i.e. atlas-application.properties) or 
> > something that can be configured by Atlas consumers - not a compile-time 
> > mechanism like Guice.  Also, for our use cases, it would be strongly 
> > preferable to enable hard delete at the API and/or type level and not be 
> > limited to a system-wide configuration.

I will make the delete configurable using application properties. But hard 
delete/soft delete at API/type level has to be done in another jira


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128551
---


On April 13, 2016, 4:12 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 13, 2016, 4:12 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1156b67 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS



> On April 13, 2016, 10:49 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java,
> >  line 42
> > 
> >
> > Should we keep timestamp same for the entire delete op?

Added to RequestContext as requestTime which is set at request initialisation


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128784
---


On April 13, 2016, 4:12 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 13, 2016, 4:12 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1156b67 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
> 1a0fa20 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 7857483 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Struct.java 70deab2 
>   
>

Re: Review Request 46011: ATLAS-622 Introduce soft delete

2016-04-14 Thread Shwetha GS



> On April 13, 2016, 9:13 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java,
> >  line 67
> > 
> >
> > when is state being cleared?

In org.apache.atlas.web.filters.AuditFilter which is called for every request


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128746
---


On April 13, 2016, 4:12 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 13, 2016, 4:12 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  f007a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a0f1d40 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> dbba926 
>   client/src/main/java/org/apache/atlas/AtlasClient.java cc87706 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 75d14f0 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1156b67 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
> 1a0fa20 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 7857483 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Struct.java 70deab2 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/DownCastStructInstance.java
>

[jira] [Created] (ATLAS-668) GraphHelper.addEdge() should add an edge only if there is no existing edge

2016-04-14 Thread Shwetha G S (JIRA)

Shwetha G S created ATLAS-668:
-

 Summary: GraphHelper.addEdge() should add an edge only if there is 
no existing edge
 Key: ATLAS-668
 URL: https://issues.apache.org/jira/browse/ATLAS-668
 Project: Atlas
  Issue Type: Bug
Reporter: Shwetha G S


Currently, addEdge adds edge always and because of any code issues, we might 
end with duplicate edges. addEdge should add a check before adding edge



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-667) Entity delete should check for required reverse references

2016-04-14 Thread Shwetha G S (JIRA)

Shwetha G S created ATLAS-667:
-

 Summary: Entity delete should check for required reverse references
 Key: ATLAS-667
 URL: https://issues.apache.org/jira/browse/ATLAS-667
 Project: Atlas
  Issue Type: Bug
Reporter: Shwetha G S
 Fix For: 0.7-incubating


When an entity is deleted, need to make sure that there are no references to 
the deleted entity with required attribute. For example, if hive_table has a 
required attribute db of type hive_database, then db entity can't be deleted if 
there is a table referencing it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-532) Change Data types of all timestamps(currently long)

2016-04-14 Thread Hemanth Yamijala (JIRA)


[ 
https://issues.apache.org/jira/browse/ATLAS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240824#comment-15240824
 ] 

Hemanth Yamijala commented on ATLAS-532:


A couple of comments / questions:

* We have defined constants for createTime etc in {{HiveMetaStoreBridge}}. It 
would be really nice if we could define these in {{HiveDataModelGenerator}} and 
use them both there and in other referring places. Similarly for the process 
related attributes.
* In {{HiveHookIT.verifyTimestamps}}, there is an extraneous line: 
{{TypeSystem.getInstance().getDateFormat().parse(createTimeStr);}}. I don't 
think this is required, as it is being assigned to a variable before.
* I saw that in Falcon and Storm we are still referring to timestamps as long 
datatypes. (Sqoop and HDFS seem fine). Ideally we would want to change those 
too. We could certainly take these as part of another JIRA. Could you please 
file that, if we take that path?
* Would the API start serializing these differently now? If yes, any impact on 
existing apps including Åtlas's own UI that we need to call out? (Maybe mark 
these as incompatible changes and announce on the dev list etc)


> Change Data types of all timestamps(currently long)
> ---
>
> Key: ATLAS-532
> URL: https://issues.apache.org/jira/browse/ATLAS-532
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 0.6-incubating, 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-532.1.patch, ATLAS-532.2.patch, ATLAS-532.patch
>
>
> startTime, endTime, createTime to date



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New Committer: Hemanth Yamijala

2016-04-14 Thread Venkata R Madugundu


Congrats Hemanth.



From:   Seetharam Venkatesh 
To: dev@atlas.incubator.apache.org
Date:   14/04/16 11:26 AM
Subject:Re: [ANNOUNCE] New Committer: Hemanth Yamijala



Thanks Shwetha for doing the honors and Congratulations Hemanth!

On Wed, Apr 13, 2016 at 10:15 PM Shwetha Shivalingamurthy <
sshivalingamur...@hortonworks.com> wrote:

> Hi All,
>
> The Apache Atlas PPMC has asked Hemanth Yamijala to become a committer
and
> we are pleased to announce that he has accepted.
>
> Regards,
> Shwetha
>
>

54 matches

Mail list logo