[jira] [Commented] (ATLAS-22) QuickStart should use hive data model instead of its own

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14805032#comment-14805032
 ] 

Shwetha G S commented on ATLAS-22:
--

We should probably use a different model for quick start

> QuickStart should use hive data model instead of its own
> 
>
> Key: ATLAS-22
> URL: https://issues.apache.org/jira/browse/ATLAS-22
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 0.5-incubating
>Reporter: Venkatesh Seetharam
>Priority: Minor
> Fix For: 0.5.1-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-132) Search Controller Optimization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876854#comment-14876854
 ] 

ASF GitHub Bot commented on ATLAS-132:
--

Github user AnilGayakwad closed the pull request at:

https://github.com/apache/incubator-atlas/pull/11


> Search Controller Optimization
> --
>
> Key: ATLAS-132
> URL: https://issues.apache.org/jira/browse/ATLAS-132
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 0.5.1-incubating
>Reporter: Erik Bergenholtz
>Assignee: Rohit
> Fix For: 0.6-incubating
>
> Attachments: ATLAS-132-v0.patch
>
>
> Search Controller currently makes multiple calls. This can and should be 
> optimized as a single API call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

> On Sept. 18, 2015, 5:29 a.m., Suma Shivaprasad wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java,
> >  line 296
> > 
> >
> > is sdQualifiedName supposed to be same as tableQualifiedName ?

sd is part of both table and index. For table, sdQualifiedName = 
tableQualifiedName. For index, sdQualifiedName = indexQualifiedName


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99486
---


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> 

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99486
---



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 (line 277)


is sdQualifiedName supposed to be same as tableQualifiedName ?


- Suma Shivaprasad


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 1b0faa5 
>   
> 

Re: Review Request 38341: Provide Entity Change Notification

[Shwetha GS]

> On Sept. 15, 2015, 5:44 a.m., Shwetha GS wrote:
> > notification/src/main/java/org/apache/atlas/notification/entity/EntityChangeConsumer.java,
> >  line 41
> > 
> >
> > why expose interface using threadpool and listeners. Why not expose 
> > NotificationInterface directly.
> > 
> > Exposing through threadpool and listener creates following issues:
> > 1. Users may not be aware of threadpool created on client side. This 
> > may add to debugging complexity. They may even create their own thread pool 
> > as well.
> > 2. If users have another way of parallelism or another thread pool of 
> > their own, they can't combine both.
> > 3. Listeners are called synchronously. Users may not be ok with it
> > 
> > 
> > In general, its better to provide thin clients to users so that they 
> > can implement their own usecases their own way. That way, maintenance on 
> > our end will be less. Makes sense?
> 
> Tom Beerbower wrote:
> I see your point and don't disagree but there is no requirement for the 
> user to use this class.  With the code as it currently is, they can inject a 
> NotificationInterface, create the consumers themselves and manage their own 
> threads.  My main intent with introducing this class was to give the users an 
> easy to use interface out of the box.  It really just saves them from having 
> to write the same code.  
> 
> Also, I basically copied most of the EntityChangeConsumer code from the 
> existing NotificationHookConsumer code, which does the same as far as 
> threadpool, etc.  Don't your arguments apply there as well?
> 
> Shwetha GS wrote:
> The problem with giving more in client libraries is that it goes as a 
> contract and it needs to be perfect, should work for all users, make sure 
> that its backward compatible in all future releases. So, more maintenance. 
> Its better to give the bare minimum and let users build on that. 
> NotificationInterafce is not complicated, just create consumer and they get 
> an iterator on message. If you think thats complicated, simplify it.
> 
> As for using the same in NotificationHookConsumer, 
> NotificationHookConsumer is part of atlas server. Currently, atlas server 
> doesn't have a thread pool and hence we need it. We can change it anytime 
> without informing the users(not the admins who monitor the atlas service). 
> Its part of server, hence not all users need to debug this(only the admins) 
> and typically, all services will have thread pools.
> 
> Tom Beerbower wrote:
> I'm not arguing that NotificationInterface is complicated.  I just 
> thought that the EntityChangeConsumer was in line with the work you had 
> already done in NotificationHookConsumer and was a bit easier to use than 
> creating the consumers directly.  I'll remove EntityChangeConsumer.
> 
> Aren't the other services that implement a hook just clients of Atlas 
> hook notification, just like Ranger is a client of Atlas entity notification? 
>  Maybe I don't understand correctly how the hooks work.
> 
> Tom Beerbower wrote:
> Okay, I think that I see now.  It's flipped from the entity 
> notification... the hook is the sender and the server is the consumer.  
> 
> I don't see any usages of NotificationHookConsumer?  Can you explain how 
> it will be used (where is start called)?

NotificationHookConsumer is a service thats started and stopped with atlas 
start and stop. Its part of ATLAS-58 patch


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38341/#review98990
---


On Sept. 14, 2015, 4:08 a.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38341/
> ---
> 
> (Updated Sept. 14, 2015, 4:08 a.m.)
> 
> 
> Review request for atlas, John Speidel and Shwetha GS.
> 
> 
> Bugs: ATLAS-158
> https://issues.apache.org/jira/browse/ATLAS-158
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add entity change notification to Atlas based using the existing 
> atlas-notification module.
> 
> First cut at a patch for the Atlas entity change notification.  Note that at 
> a minimum additional unit tests are required.  I'm putting up the review to 
> get some initial feedback.
> 
> 
> Diffs
> -
> 
>   notification/src/main/java/org/apache/atlas/notification/entity/Entity.java 
> PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/EntityChangeConsumer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/EntityChangeListener.java
>  PRE-CREATION 
>   
> 

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99487
---



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
267)


Can we make this info ? This migght be the case of delays in synchronous 
calls and would be good to have visibility.



client/src/main/java/org/apache/atlas/AtlasClient.java (line 109)


we should make rge readTimeout configurable and also add connectTimeout



common/src/main/java/org/apache/atlas/service/Services.java (line 38)


should we add a register and unregister service method here?


- Suma Shivaprasad


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99499
---



notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java (line 
131)


should we throw and exceptiopn instead?


- Suma Shivaprasad


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 1b0faa5 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  cc10ea3 
>   

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

> On Sept. 18, 2015, 6:24 a.m., Suma Shivaprasad wrote:
> > client/src/main/java/org/apache/atlas/AtlasClient.java, line 110
> > 
> >
> > we should make rge readTimeout configurable and also add connectTimeout

read timeout is already configurable. will add connect timeout


> On Sept. 18, 2015, 6:24 a.m., Suma Shivaprasad wrote:
> > common/src/main/java/org/apache/atlas/service/Services.java, line 38
> > 
> >
> > should we add a register and unregister service method here?

register/unregister is done using guice in NotificationModule


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99487
---


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99522
---



notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java (line 
131)


some configs like kafka endpoint don't have protocol. But I need URL object 
to get ports - don't want to parse it myself


- Shwetha GS


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 1b0faa5 
>   
> 

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
---

(Updated Sept. 18, 2015, 11:14 a.m.)


Review request for atlas.


Changes
---

fixed review comments. added thread pool for hive hook


Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58


Repository: atlas


Description
---

Hive hook sends notification messages (list of entities). The Notification 
consumer on server side consumes these messages and registers the entities. The 
server handles de-duping of entities based on the unique attribute of the entity
Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the 
entity. If entity doesn't have any unique attribute, de-duping is not done and 
new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity 
(required for hive hook) - backward incompatible
4. Moved submit and list from EntityResource to EntitiesResource - backward 
incompatible 
5. Moved security tests from integration tests to unit tests - as they were 
creating issues with server start as jetty already starts another server for 
integration tests
6. Removed some duplicate tests from repository module (the same tests exist in 
typesystem module as well)
7. In webapp ITs, re-used the types defined
8. Hive hook now sends notifications instead of registering entities. Sending 
notification is done synchronously. So, this adds to hive command execution 
delay. But this also makes it reliable

Pending:
1. Entity updates like alter table commands are not handlded. Will create 
another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
their own. Still checking on this


Diffs (updated)
-

  addons/hive-bridge/pom.xml 914d8c6 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 4d1af4e 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
fe26446 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 cb8359e 
  addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
2bf3aba 
  client/pom.xml 2e27930 
  client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
  client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
  client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1 
  client/src/test/resources/application.properties dbd6002 
  common/pom.xml PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
  dashboard/public/modules/details/detailsResource.js af2f249 
  distro/pom.xml d42d78f 
  distro/src/conf/application.properties bf323a7 
  distro/src/main/assemblies/standalone-package.xml 625396a 
  docs/pom.xml 4535fe0 
  docs/src/site/resources/architecture.png 826df37 
  docs/src/site/resources/data-types.png 3aa1904 
  docs/src/site/resources/guide-class-diagram.png ca51239 
  docs/src/site/resources/guide-instance-graph.png a2c8f82 
  docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
  docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
  docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
  docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
  docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
  docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION 
  docs/src/site/resources/types-instance.png 6afca21 
  docs/src/site/site.xml b412569 
  docs/src/site/twiki/Architecture.twiki 5f10fde 
  docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION 
  docs/src/site/twiki/Configuration.twiki e2132cf 
  docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
  docs/src/site/twiki/index.twiki 40d2350 
  notification/pom.xml b036855 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
9978275 
  
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
 PRE-CREATION 
  
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 36a62f0 
  
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 0951124 
  
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
 db17e35 
  notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
02752dc 
  pom.xml e8d18a2 
  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
3d47293 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
fbd01de 
  

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Shwetha GS]

> On Sept. 18, 2015, 7:04 a.m., Suma Shivaprasad wrote:
> > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java, 
> > line 139
> > 
> >
> > should we throw and exceptiopn instead?

some configs like kafka endpoint don't have protocol. But I need URL object to 
get ports - don't want to parse it myself


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99499
---


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java
>  PRE-CREATION 
>