Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

> On Sept. 21, 2015, 3:36 p.m., Suma Shivaprasad wrote:
> > docs/src/site/twiki/TypeSystem.twiki, line 8
> > 
> >
> > pls add docs for modelling unique entities by qualifiedName

I have added in EntitiesResource API. Currently, there is no documentation at 
all on entity creation. It will be a big effort otherwise


> On Sept. 21, 2015, 3:36 p.m., Suma Shivaprasad wrote:
> > typesystem/src/main/resources/application.properties, line 61
> > 
> >
> > pls add comments for the configs

Its already in configuration twiki


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99753
---


On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 21, 2015, 8:13 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
>   

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99954
---

Ship it!


Ship It!

- Suma Shivaprasad


On Sept. 22, 2015, 9:09 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 22, 2015, 9:09 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   distro/src/test/python/scripts/TestMetadata.py 57f247f 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION 
>   docs/src/site/resources/types-instance.png 6afca21 
>   docs/src/site/site.xml b412569 
>   docs/src/site/twiki/Architecture.twiki 5f10fde 
>   docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION 
>   docs/src/site/twiki/Configuration.twiki e2132cf 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   docs/src/site/twiki/QuickStart.twiki d899f95 
>   docs/src/site/twiki/TypeSystem.twiki 78c3503 
>   docs/src/site/twiki/index.twiki 40d2350 
>   notification/pom.xml 

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

> On Sept. 21, 2015, 3:22 p.m., Suma Shivaprasad wrote:
> > typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java, line 
> > 32
> > 
> >
> > have we moved this from client to ypesystem?

yes


> On Sept. 21, 2015, 3:22 p.m., Suma Shivaprasad wrote:
> > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java,
> >  line 121
> > 
> >
> > if possible, can you pls fix the error message in other places where 
> > ValueCnversionException is being thrown curently :) Right now, we dont ge 
> > to know which attribute is causing the issue

In the validation of actual attribute, the attribute name is not available. All 
attribute validation are called from Struct. So, this captures everything


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99752
---


On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 21, 2015, 8:13 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
---

(Updated Sept. 22, 2015, 9:09 a.m.)


Review request for atlas.


Changes
---

Addressed review comments


Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58


Repository: atlas


Description
---

Hive hook sends notification messages (list of entities). The Notification 
consumer on server side consumes these messages and registers the entities. The 
server handles de-duping of entities based on the unique attribute of the entity
Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the 
entity. If entity doesn't have any unique attribute, de-duping is not done and 
new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity 
(required for hive hook) - backward incompatible
4. Moved submit and list from EntityResource to EntitiesResource - backward 
incompatible 
5. Moved security tests from integration tests to unit tests - as they were 
creating issues with server start as jetty already starts another server for 
integration tests
6. Removed some duplicate tests from repository module (the same tests exist in 
typesystem module as well)
7. In webapp ITs, re-used the types defined
8. Hive hook now sends notifications instead of registering entities. Sending 
notification is done synchronously. So, this adds to hive command execution 
delay. But this also makes it reliable

Pending:
1. Entity updates like alter table commands are not handlded. Will create 
another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
their own. Still checking on this


Diffs (updated)
-

  addons/hive-bridge/pom.xml 914d8c6 
  addons/hive-bridge/src/bin/import-hive.sh 99300e8 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 4d1af4e 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
fe26446 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 cb8359e 
  addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
2bf3aba 
  client/pom.xml 2e27930 
  client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
  client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
  client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1 
  client/src/test/resources/application.properties dbd6002 
  common/pom.xml PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
  dashboard/public/modules/details/detailsResource.js af2f249 
  dashboard/public/modules/search/searchController.js b5e6c37 
  distro/pom.xml d42d78f 
  distro/src/bin/atlas_start.py 7680f04 
  distro/src/conf/application.properties bf323a7 
  distro/src/main/assemblies/standalone-package.xml 625396a 
  distro/src/test/python/scripts/TestMetadata.py 57f247f 
  docs/pom.xml 4535fe0 
  docs/src/site/resources/architecture.png 826df37 
  docs/src/site/resources/data-types.png 3aa1904 
  docs/src/site/resources/guide-class-diagram.png ca51239 
  docs/src/site/resources/guide-instance-graph.png a2c8f82 
  docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
  docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
  docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
  docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
  docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
  docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION 
  docs/src/site/resources/types-instance.png 6afca21 
  docs/src/site/site.xml b412569 
  docs/src/site/twiki/Architecture.twiki 5f10fde 
  docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION 
  docs/src/site/twiki/Configuration.twiki e2132cf 
  docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
  docs/src/site/twiki/QuickStart.twiki d899f95 
  docs/src/site/twiki/TypeSystem.twiki 78c3503 
  docs/src/site/twiki/index.twiki 40d2350 
  notification/pom.xml b036855 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
9978275 
  
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 36a62f0 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 0951124 
  
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
 db17e35 

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99747
---



repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
 (line 647)


we should rename this function to make the intention clear



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 197)


we should include the typedefinition as well in the exception?



repository/src/main/java/org/apache/atlas/services/MetadataService.java (line 
83)


pls add java doc for th args to make it clear


- Suma Shivaprasad


On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 21, 2015, 8:13 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
>   

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99753
---



docs/src/site/twiki/TypeSystem.twiki (line 8)


pls add docs for modelling unique entities by qualifiedName



typesystem/src/main/resources/application.properties (line 59)


pls add comments for the configs


- Suma Shivaprasad


On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 21, 2015, 8:13 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION 
>   docs/src/site/resources/types-instance.png 6afca21 
>   docs/src/site/site.xml b412569 
>   docs/src/site/twiki/Architecture.twiki 5f10fde 
>   docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION 
>   

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99752
---



typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java (line 32)


have we moved this from client to ypesystem?



typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java
 (line 121)


if possible, can you pls fix the error message in other places where 
ValueCnversionException is being thrown curently :) Right now, we dont ge to 
know which attribute is causing the issue


- Suma Shivaprasad


On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 21, 2015, 8:13 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION 
>   

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

> On Sept. 18, 2015, 5:29 a.m., Suma Shivaprasad wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java,
> >  line 296
> > 
> >
> > is sdQualifiedName supposed to be same as tableQualifiedName ?

sd is part of both table and index. For table, sdQualifiedName = 
tableQualifiedName. For index, sdQualifiedName = indexQualifiedName


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99486
---


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> 

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99486
---



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 (line 277)


is sdQualifiedName supposed to be same as tableQualifiedName ?


- Suma Shivaprasad


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 1b0faa5 
>   
> 

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99487
---



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
267)


Can we make this info ? This migght be the case of delays in synchronous 
calls and would be good to have visibility.



client/src/main/java/org/apache/atlas/AtlasClient.java (line 109)


we should make rge readTimeout configurable and also add connectTimeout



common/src/main/java/org/apache/atlas/service/Services.java (line 38)


should we add a register and unregister service method here?


- Suma Shivaprasad


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99499
---



notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java (line 
131)


should we throw and exceptiopn instead?


- Suma Shivaprasad


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 1b0faa5 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  cc10ea3 
>   

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

> On Sept. 18, 2015, 6:24 a.m., Suma Shivaprasad wrote:
> > client/src/main/java/org/apache/atlas/AtlasClient.java, line 110
> > 
> >
> > we should make rge readTimeout configurable and also add connectTimeout

read timeout is already configurable. will add connect timeout


> On Sept. 18, 2015, 6:24 a.m., Suma Shivaprasad wrote:
> > common/src/main/java/org/apache/atlas/service/Services.java, line 38
> > 
> >
> > should we add a register and unregister service method here?

register/unregister is done using guice in NotificationModule


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99487
---


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99522
---



notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java (line 
131)


some configs like kafka endpoint don't have protocol. But I need URL object 
to get ports - don't want to parse it myself


- Shwetha GS


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 1b0faa5 
>   
> 

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
---

(Updated Sept. 18, 2015, 11:14 a.m.)


Review request for atlas.


Changes
---

fixed review comments. added thread pool for hive hook


Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58


Repository: atlas


Description
---

Hive hook sends notification messages (list of entities). The Notification 
consumer on server side consumes these messages and registers the entities. The 
server handles de-duping of entities based on the unique attribute of the entity
Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the 
entity. If entity doesn't have any unique attribute, de-duping is not done and 
new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity 
(required for hive hook) - backward incompatible
4. Moved submit and list from EntityResource to EntitiesResource - backward 
incompatible 
5. Moved security tests from integration tests to unit tests - as they were 
creating issues with server start as jetty already starts another server for 
integration tests
6. Removed some duplicate tests from repository module (the same tests exist in 
typesystem module as well)
7. In webapp ITs, re-used the types defined
8. Hive hook now sends notifications instead of registering entities. Sending 
notification is done synchronously. So, this adds to hive command execution 
delay. But this also makes it reliable

Pending:
1. Entity updates like alter table commands are not handlded. Will create 
another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
their own. Still checking on this


Diffs (updated)
-

  addons/hive-bridge/pom.xml 914d8c6 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 4d1af4e 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
fe26446 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 cb8359e 
  addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
2bf3aba 
  client/pom.xml 2e27930 
  client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
  client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
  client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1 
  client/src/test/resources/application.properties dbd6002 
  common/pom.xml PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
  dashboard/public/modules/details/detailsResource.js af2f249 
  distro/pom.xml d42d78f 
  distro/src/conf/application.properties bf323a7 
  distro/src/main/assemblies/standalone-package.xml 625396a 
  docs/pom.xml 4535fe0 
  docs/src/site/resources/architecture.png 826df37 
  docs/src/site/resources/data-types.png 3aa1904 
  docs/src/site/resources/guide-class-diagram.png ca51239 
  docs/src/site/resources/guide-instance-graph.png a2c8f82 
  docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
  docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
  docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
  docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
  docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
  docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION 
  docs/src/site/resources/types-instance.png 6afca21 
  docs/src/site/site.xml b412569 
  docs/src/site/twiki/Architecture.twiki 5f10fde 
  docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION 
  docs/src/site/twiki/Configuration.twiki e2132cf 
  docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
  docs/src/site/twiki/index.twiki 40d2350 
  notification/pom.xml b036855 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
9978275 
  
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 36a62f0 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 0951124 
  
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
 db17e35 
  notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
02752dc 
  pom.xml e8d18a2 
  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
3d47293 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
fbd01de 
  

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

> On Sept. 18, 2015, 7:04 a.m., Suma Shivaprasad wrote:
> > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java, 
> > line 139
> > 
> >
> > should we throw and exceptiopn instead?

some configs like kafka endpoint don't have protocol. But I need URL object to 
get ports - don't want to parse it myself


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99499
---


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java
>  PRE-CREATION 
>   

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
---

(Updated Sept. 15, 2015, 6:33 a.m.)


Review request for atlas.


Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58


Repository: atlas


Description (updated)
---

Hive hook sends notification messages (list of entities). The Notification 
consumer on server side consumes these messages and registers the entities. The 
server handles de-duping of entities based on the unique attribute of the entity

Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the 
entity. If entity doesn't have any unique attribute, de-duping is not done and 
new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity 
(required for hive hook)
4. Moved security tests from integration tests to unit tests - as they were 
creating issues with server start as jetty already starts another server for 
integration tests
5. Removed some duplicate tests from repository module (the same tests exist in 
typesystem module as well)
6. In webapp ITs, re-used the types defined
7. Hive hook now sends notifications instead of registering entities. Sending 
notification is done synchronously. So, this adds to hive command execution 
delay. But this also makes it reliable

Pending:
1. Entity updates like alter table commands are not handlded. Will create 
another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
their own. Still checking on this


Diffs
-

  addons/hive-bridge/pom.xml 914d8c6 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 4d1af4e 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
fe26446 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 cb8359e 
  addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
2bf3aba 
  client/pom.xml 2e27930 
  client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
  client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
  client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1 
  client/src/test/resources/application.properties dbd6002 
  common/pom.xml PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
  dashboard/public/modules/details/detailsResource.js af2f249 
  distro/pom.xml d42d78f 
  distro/src/conf/application.properties bf323a7 
  docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
  notification/pom.xml b036855 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
9978275 
  
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 36a62f0 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 0951124 
  
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
 db17e35 
  notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
02752dc 
  pom.xml e8d18a2 
  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
3d47293 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
fbd01de 
  repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
f58d6de 
  
repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
1b0faa5 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
 cc10ea3 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
ceb8a65 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
56168db 
  repository/src/main/java/org/apache/atlas/services/MetadataService.java 
8f82f88 
  repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 
18b1d1b 
  repository/src/test/java/org/apache/atlas/TestUtils.java de41793 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 127e6d7 
  
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 
2c7d61a 
  repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
 487ac84 
  

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
---

(Updated Sept. 15, 2015, 6:50 a.m.)


Review request for atlas.


Changes
---

Fixed hive-bridge IT failures


Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58


Repository: atlas


Description
---

Hive hook sends notification messages (list of entities). The Notification 
consumer on server side consumes these messages and registers the entities. The 
server handles de-duping of entities based on the unique attribute of the entity

Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the 
entity. If entity doesn't have any unique attribute, de-duping is not done and 
new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity 
(required for hive hook)
4. Moved security tests from integration tests to unit tests - as they were 
creating issues with server start as jetty already starts another server for 
integration tests
5. Removed some duplicate tests from repository module (the same tests exist in 
typesystem module as well)
6. In webapp ITs, re-used the types defined
7. Hive hook now sends notifications instead of registering entities. Sending 
notification is done synchronously. So, this adds to hive command execution 
delay. But this also makes it reliable

Pending:
1. Entity updates like alter table commands are not handlded. Will create 
another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
their own. Still checking on this


Diffs (updated)
-

  addons/hive-bridge/pom.xml 914d8c6 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 4d1af4e 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
fe26446 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 cb8359e 
  addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
2bf3aba 
  client/pom.xml 2e27930 
  client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
  client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
  client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1 
  client/src/test/resources/application.properties dbd6002 
  common/pom.xml PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
  dashboard/public/modules/details/detailsResource.js af2f249 
  distro/pom.xml d42d78f 
  distro/src/conf/application.properties bf323a7 
  docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
  notification/pom.xml b036855 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
9978275 
  
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 36a62f0 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 0951124 
  
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
 db17e35 
  notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
02752dc 
  pom.xml e8d18a2 
  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
3d47293 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
fbd01de 
  repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
f58d6de 
  
repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
1b0faa5 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
 cc10ea3 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
ceb8a65 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
56168db 
  repository/src/main/java/org/apache/atlas/services/MetadataService.java 
8f82f88 
  repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 
18b1d1b 
  repository/src/test/java/org/apache/atlas/TestUtils.java de41793 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 127e6d7 
  
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 
2c7d61a 
  repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37