[jira] [Reopened] (ATLAS-871) Make audit repository implementation configurable

[Shwetha G S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S reopened ATLAS-871:
---

> Make audit repository implementation configurable
> -
>
> Key: ATLAS-871
> URL: https://issues.apache.org/jira/browse/ATLAS-871
> Project: Atlas
>  Issue Type: Task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-880) Audit log: Tag association to hive column is not logged

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322003#comment-15322003
 ] 

Shwetha G S commented on ATLAS-880:
---

Since column is an entity, audit is logged under column entity. Table is 
referenced entity in this case, and audit events of column can't go under table 
entity

> Audit log: Tag association to hive column is not logged
> ---
>
> Key: ATLAS-880
> URL: https://issues.apache.org/jira/browse/ATLAS-880
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Shwetha G S
> Fix For: 0.7-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (ATLAS-880) Audit log: Tag association to hive column is not logged

[Shwetha G S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S resolved ATLAS-880.
---
Resolution: Not A Problem

> Audit log: Tag association to hive column is not logged
> ---
>
> Key: ATLAS-880
> URL: https://issues.apache.org/jira/browse/ATLAS-880
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Shwetha G S
> Fix For: 0.7-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 48476: ATLAS-642 import-hive should create the lineage for external tables

[Vimal Sharma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48476/
---

Review request for atlas.


Bugs: ATLAS-642
https://issues.apache.org/jira/browse/ATLAS-642


Repository: atlas


Description
---

import-hive.sh script should create the lineage for external tables


Diffs
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 fe07d73 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
9026b3c 

Diff: https://reviews.apache.org/r/48476/diff/


Testing
---

Yes


Thanks,

Vimal Sharma

[jira] [Created] (ATLAS-886) Improve association of terms with entity (Show tree view)

[Keval Bhatt (JIRA)]

Keval Bhatt created ATLAS-886:
-

 Summary: Improve association of terms with entity (Show tree view)
 Key: ATLAS-886
 URL: https://issues.apache.org/jira/browse/ATLAS-886
 Project: Atlas
  Issue Type: Improvement
Reporter: Keval Bhatt
Assignee: Keval Bhatt


Currently, UI shows hierarchical name Terms in the popup, which might be 
confusing for end users if they have long list of terms. Need to improve 
association of terms with entity and Show tree view so that users can traverse 
through the tree of Terms to select a term. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-642) import-hive should create the lineage for external tables

[Vimal Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321952#comment-15321952
 ] 

Vimal Sharma commented on ATLAS-642:


[~suma.shivaprasad]
It makes sense to use the same function getProcessQualifiedName() in HiveHook 
to get the process qualified name. However we need "SHOW CREATE TABLE" query to 
get the same qualified name as in HiveHook. The reason is that if qualified 
name differs then import_hive.sh would end up creating a new process and we 
will have two different lineage graphs for the same table.

Process Referenceable creation in HiveHook uses a HiveEvent to get the process 
details. We don't have a HiveEvent in HMSB.

> import-hive should create the lineage for external tables
> -
>
> Key: ATLAS-642
> URL: https://issues.apache.org/jira/browse/ATLAS-642
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>Assignee: Vimal Sharma
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-642.patch
>
>
> import-hive can also start showing lineage between HDFS paths and tables. 
> However the create table process should also register the exact query that 
> was  used to create the table. Show create table does that by reconstructing 
> the query. Need to figure out if we can potentially use that to create the 
> query



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-885) optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 seconds

[Hemanth Yamijala (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321942#comment-15321942
 ] 

Hemanth Yamijala commented on ATLAS-885:


Nope. I didn't run the search tests yet. We will run these after the commit as 
part of regular performance runs.

> optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 
> seconds
> --
>
> Key: ATLAS-885
> URL: https://issues.apache.org/jira/browse/ATLAS-885
> Project: Atlas
>  Issue Type: Bug
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
> Attachments: ATLAS-885.patch
>
>
> For every 5 seconds following logs are added to 
> /var/logs/atlas/application.log, which increases the log file size 
> unnecessarily and makes the log difficult to use.
> {noformat}
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Process 
> identifier=hconnection-0x7878431f connecting to ZooKeeper 
> ensemble=localhost:2181 (RecoverableZooKeeper:120)
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Initiating client 
> connection, connectString=localhost:2181 sessionTimeout=9 
> watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@78b60e6a 
> (ZooKeeper:438)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Opening socket 
> connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt 
> to authenticate using SASL (unknown error) (ClientCnxn:1019)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Socket connection 
> established to localhost.localdomain/127.0.0.1:2181, initiating session 
> (ClientCnxn:864)
> 2016-05-17 22:01:18,531 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Session 
> establishment complete on server localhost.localdomain/127.0.0.1:2181, 
> sessionid = 0x154c1cf558e18b9, negotiated timeout = 4 (ClientCnxn:1279)
> 2016-05-17 22:01:18,539 INFO - [pool-5-thread-1:] ~ Closing zookeeper 
> sessionid=0x154c1cf558e18b9 (ConnectionManager$HConnectionImplementation:1684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1:] ~ Session: 
> 0x154c1cf558e18b9 closed (ZooKeeper:684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1-EventThread:] ~ EventThread 
> shut down (ClientCnxn:524)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-885) optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 seconds

[Madhan Neethiraj (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321912#comment-15321912
 ] 

Madhan Neethiraj commented on ATLAS-885:


[~yhemanth] thanks for the review and trying multiple use cases with the patch. 
I noticed a better search response after this patch. Did you notice any 
differences?

> optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 
> seconds
> --
>
> Key: ATLAS-885
> URL: https://issues.apache.org/jira/browse/ATLAS-885
> Project: Atlas
>  Issue Type: Bug
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
> Attachments: ATLAS-885.patch
>
>
> For every 5 seconds following logs are added to 
> /var/logs/atlas/application.log, which increases the log file size 
> unnecessarily and makes the log difficult to use.
> {noformat}
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Process 
> identifier=hconnection-0x7878431f connecting to ZooKeeper 
> ensemble=localhost:2181 (RecoverableZooKeeper:120)
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Initiating client 
> connection, connectString=localhost:2181 sessionTimeout=9 
> watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@78b60e6a 
> (ZooKeeper:438)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Opening socket 
> connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt 
> to authenticate using SASL (unknown error) (ClientCnxn:1019)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Socket connection 
> established to localhost.localdomain/127.0.0.1:2181, initiating session 
> (ClientCnxn:864)
> 2016-05-17 22:01:18,531 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Session 
> establishment complete on server localhost.localdomain/127.0.0.1:2181, 
> sessionid = 0x154c1cf558e18b9, negotiated timeout = 4 (ClientCnxn:1279)
> 2016-05-17 22:01:18,539 INFO - [pool-5-thread-1:] ~ Closing zookeeper 
> sessionid=0x154c1cf558e18b9 (ConnectionManager$HConnectionImplementation:1684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1:] ~ Session: 
> 0x154c1cf558e18b9 closed (ZooKeeper:684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1-EventThread:] ~ EventThread 
> shut down (ClientCnxn:524)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-885) optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 seconds

[Hemanth Yamijala (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321879#comment-15321879
 ] 

Hemanth Yamijala commented on ATLAS-885:


Multi-threaded tag creation also works fine.

> optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 
> seconds
> --
>
> Key: ATLAS-885
> URL: https://issues.apache.org/jira/browse/ATLAS-885
> Project: Atlas
>  Issue Type: Bug
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
> Attachments: ATLAS-885.patch
>
>
> For every 5 seconds following logs are added to 
> /var/logs/atlas/application.log, which increases the log file size 
> unnecessarily and makes the log difficult to use.
> {noformat}
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Process 
> identifier=hconnection-0x7878431f connecting to ZooKeeper 
> ensemble=localhost:2181 (RecoverableZooKeeper:120)
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Initiating client 
> connection, connectString=localhost:2181 sessionTimeout=9 
> watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@78b60e6a 
> (ZooKeeper:438)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Opening socket 
> connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt 
> to authenticate using SASL (unknown error) (ClientCnxn:1019)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Socket connection 
> established to localhost.localdomain/127.0.0.1:2181, initiating session 
> (ClientCnxn:864)
> 2016-05-17 22:01:18,531 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Session 
> establishment complete on server localhost.localdomain/127.0.0.1:2181, 
> sessionid = 0x154c1cf558e18b9, negotiated timeout = 4 (ClientCnxn:1279)
> 2016-05-17 22:01:18,539 INFO - [pool-5-thread-1:] ~ Closing zookeeper 
> sessionid=0x154c1cf558e18b9 (ConnectionManager$HConnectionImplementation:1684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1:] ~ Session: 
> 0x154c1cf558e18b9 closed (ZooKeeper:684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1-EventThread:] ~ EventThread 
> shut down (ClientCnxn:524)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-885) optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 seconds

[Hemanth Yamijala (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321867#comment-15321867
 ] 

Hemanth Yamijala commented on ATLAS-885:


Looks good to me. The change to use existing connection to get HTable instance 
is clean. The getDeployment() call is deeply nested inside Titan code. I tried 
to see if I could make sense of it, but couldn't. So, I'd say let's go ahead 
and watch out for any issues.

I did run a micro benchmark locally. Creating 1000 tables on my local machine 
does not change timing characteristics (around 4 minutes) either with or 
without the patch.

One more thing is that I don't see these log lines printed with my version of 
HBase client. (The base HBase version is the same between internal HDP and 
external HBase I am using - 1.1.2, so I don't know where the difference is 
coming from. Either way, seems like an improvement - so +1 from me.

> optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 
> seconds
> --
>
> Key: ATLAS-885
> URL: https://issues.apache.org/jira/browse/ATLAS-885
> Project: Atlas
>  Issue Type: Bug
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
> Attachments: ATLAS-885.patch
>
>
> For every 5 seconds following logs are added to 
> /var/logs/atlas/application.log, which increases the log file size 
> unnecessarily and makes the log difficult to use.
> {noformat}
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Process 
> identifier=hconnection-0x7878431f connecting to ZooKeeper 
> ensemble=localhost:2181 (RecoverableZooKeeper:120)
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Initiating client 
> connection, connectString=localhost:2181 sessionTimeout=9 
> watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@78b60e6a 
> (ZooKeeper:438)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Opening socket 
> connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt 
> to authenticate using SASL (unknown error) (ClientCnxn:1019)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Socket connection 
> established to localhost.localdomain/127.0.0.1:2181, initiating session 
> (ClientCnxn:864)
> 2016-05-17 22:01:18,531 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Session 
> establishment complete on server localhost.localdomain/127.0.0.1:2181, 
> sessionid = 0x154c1cf558e18b9, negotiated timeout = 4 (ClientCnxn:1279)
> 2016-05-17 22:01:18,539 INFO - [pool-5-thread-1:] ~ Closing zookeeper 
> sessionid=0x154c1cf558e18b9 (ConnectionManager$HConnectionImplementation:1684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1:] ~ Session: 
> 0x154c1cf558e18b9 closed (ZooKeeper:684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1-EventThread:] ~ EventThread 
> shut down (ClientCnxn:524)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48452: ATLAS-885: optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 seconds

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48452/#review136754
---


Ship it!




Ship It!

- Suma Shivaprasad


On June 9, 2016, 12:41 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48452/
> ---
> 
> (Updated June 9, 2016, 12:41 a.m.)
> 
> 
> Review request for atlas, enis and Abhay Kulkarni.
> 
> 
> Bugs: ATLAS-885
> https://issues.apache.org/jira/browse/ATLAS-885
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 
> seconds
> 
> 
> Diffs
> -
> 
>   
> titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseStoreManager.java
>  52f28af 
>   
> titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable0_98.java 
> 4ddb2f0 
>   
> titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable1_0.java 
> 5085abb 
>   
> titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/TableMask.java 
> dd3d61e 
> 
> Diff: https://reviews.apache.org/r/48452/diff/
> 
> 
> Testing
> ---
> 
> Verified that the logs generated during HTable instantiation are no more 
> present after this fix.
> Verified that basic scenarios like entity creation (via Hive hook), tag 
> creation, tag assocation, UI interaction work well.
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

[jira] [Updated] (ATLAS-641) Lineage for a view created from a view seems to be confusing(from user's perspective).

[Suma Shivaprasad (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suma Shivaprasad updated ATLAS-641:
---
Fix Version/s: (was: trunk)

> Lineage for a view created from a view seems to be confusing(from user's 
> perspective).
> --
>
> Key: ATLAS-641
> URL: https://issues.apache.org/jira/browse/ATLAS-641
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Ayub Khan
>
> Lineage for a table created from the view seems to be confusing(from user's 
> perspective).
> Steps to reproduce:
> {noformat}
> 0: jdbc:hive2://localhost:1/default>
> 0: jdbc:hive2://localhost:1/default> create table src (x int, y int, s 
> string);
> No rows affected (0.384 seconds)
> 0: jdbc:hive2://localhost:1/default> create view view1 as select * from 
> src;
> No rows affected (0.225 seconds)
> 0: jdbc:hive2://localhost:1/default> create table view_table as select * 
> from view1;
> INFO  : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO  : number of splits:1
> INFO  : Submitting tokens for job: job_local1883260823_0021
> INFO  : The url to track the job: http://localhost:8080/
> INFO  : Job running in-process (local Hadoop)
> INFO  : 2016-04-06 18:15:02,538 Stage-1 map = 100%,  reduce = 0%
> INFO  : Ended Job = job_local1883260823_0021
> INFO  : Stage-4 is selected by condition resolver.
> INFO  : Stage-3 is filtered out by condition resolver.
> INFO  : Stage-5 is filtered out by condition resolver.
> INFO  : Moving data to: 
> hdfs://localhost:9000/user/hive/warehouse/.hive-staging_hive_2016-04-06_18-14-59_156_3909183228338665119-12/-ext-10001
>  from 
> hdfs://localhost:9000/user/hive/warehouse/.hive-staging_hive_2016-04-06_18-14-59_156_3909183228338665119-12/-ext-10003
> INFO  : Moving data to: hdfs://localhost:9000/user/hive/warehouse/view_table 
> from 
> hdfs://localhost:9000/user/hive/warehouse/.hive-staging_hive_2016-04-06_18-14-59_156_3909183228338665119-12/-ext-10001
> INFO  : Table default.view_table stats: [numFiles=1, numRows=0, totalSize=0, 
> rawDataSize=0]
> No rows affected (3.8 seconds)
> 0: jdbc:hive2://localhost:1/default> alter view view1 as select * from t2;
> No rows affected (0.602 seconds)
> 0: jdbc:hive2://localhost:1/default>
> {noformat}
> Check the lineage of the resultant table(view_table).
> Link showing confusing lineage: 
> https://monosnap.com/file/qaYZcJRQnNX5BsyM12hkryv70RZK1I



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-686) Truncate table, insert into local dir, load from local dir , update, delete commands should be easily trackable from a table

[Suma Shivaprasad (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suma Shivaprasad updated ATLAS-686:
---
Fix Version/s: (was: 0.7-incubating)

> Truncate table, insert into local dir, load from local dir , update, delete 
> commands should be easily trackable from a table
> 
>
> Key: ATLAS-686
> URL: https://issues.apache.org/jira/browse/ATLAS-686
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>
> Truncate table does not show in lineage since it doesnt have any inputs. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-597) Cluster Name should be changed to cluster Id

[Suma Shivaprasad (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suma Shivaprasad updated ATLAS-597:
---
Fix Version/s: (was: 0.7-incubating)

> Cluster Name should be changed to cluster Id
> 
>
> Key: ATLAS-597
> URL: https://issues.apache.org/jira/browse/ATLAS-597
> Project: Atlas
>  Issue Type: Sub-task
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>
> Ambari Cluster Name or User specified cluster name could easily be changed by 
> users which is used in qualified names to dedup entities. For eg: hive_table 
> as  . Hence all entities will be recreated if 
> they are created again unless a mass update on repository is done with new 
> cluster name.
> Hence cluster  should be represented by an id which is constant and tied to 
> clusterName which could change



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-884) Process registration should call Entity update instead of create

[Suma Shivaprasad (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suma Shivaprasad updated ATLAS-884:
---
Attachment: ATLAS-884.patch

> Process registration should call Entity update instead of create
> 
>
> Key: ATLAS-884
> URL: https://issues.apache.org/jira/browse/ATLAS-884
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.6-incubating, 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-884.patch
>
>
> Process registration in hive hook currently calls Entity create . But should 
> call update entity instead 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: apache-atlas-nightly #307

[Apache Jenkins Server]

See 

Changes:

[sumasai.shivaprasad] ATLAS-867 Excessive logs: default log level should be set 
to 'info';

--
[...truncated 5922 lines...]
[INFO] Using incremental compilation
[INFO] Compiling 56 Java sources to 

[WARNING] Note: Some input files use or override a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[WARNING] Note: Some input files use unchecked or unsafe operations.
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ atlas-webapp 
---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 56 source files to 

[WARNING] 
:
 Some input files use or override a deprecated API.
[WARNING] 
:
 Recompile with -Xlint:deprecation for details.
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ 
atlas-webapp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- scala-maven-plugin:3.2.0:testCompile (scala-test-compile-first) @ 
atlas-webapp ---
[WARNING] Zinc server is not available at port 3030 - reverting to normal 
incremental compile
[INFO] Using incremental compilation
[INFO] Compiling 41 Java sources to 

[WARNING] Note: Some input files use unchecked or unsafe operations.
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
atlas-webapp ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 41 source files to 

[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-war-plugin:2.6:war (default-war) @ atlas-webapp ---
[INFO] Packaging webapp
[INFO] Assembling webapp [atlas-webapp] in 
[
[INFO] Processing overlay [ id org.apache.atlas:atlas-dashboardv2]
[INFO] Processing war project
[INFO] Copying webapp resources 
[
[INFO] Building jar: 

[INFO] Webapp assembled in [2644 msecs]
[INFO] Building war: 

[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
atlas-webapp ---
[INFO] 
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ atlas-webapp ---
[WARNING] The parameter forkMode is deprecated since version 2.14. Use 
forkCount and reuseForks instead.
[INFO] Surefire report directory: 

[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[WARNING] file.encoding cannot be set as system property, use 
-Dfile.encoding=... instead

---
 T E S T S
---
Running org.apache.atlas.web.service.ActiveInstanceElectorServiceTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.582 sec - in 
org.apache.atlas.web.service.ActiveInstanceElectorServiceTest
Running org.apache.atlas.web.service.CuratorFactoryTest
Tests run: 3, Failures: 0, E

Review Request 48452: ATLAS-885: optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 seconds

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48452/
---

Review request for atlas, enis and Abhay Kulkarni.


Bugs: ATLAS-885
https://issues.apache.org/jira/browse/ATLAS-885


Repository: atlas


Description
---

optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 
seconds


Diffs
-

  
titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseStoreManager.java
 52f28af 
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable0_98.java 
4ddb2f0 
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable1_0.java 
5085abb 
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/TableMask.java 
dd3d61e 

Diff: https://reviews.apache.org/r/48452/diff/


Testing
---

Verified that the logs generated during HTable instantiation are no more 
present after this fix.
Verified that basic scenarios like entity creation (via Hive hook), tag 
creation, tag assocation, UI interaction work well.


Thanks,

Madhan Neethiraj

[jira] [Updated] (ATLAS-885) optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 seconds

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-885:
---
Attachment: ATLAS-885.patch

> optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 
> seconds
> --
>
> Key: ATLAS-885
> URL: https://issues.apache.org/jira/browse/ATLAS-885
> Project: Atlas
>  Issue Type: Bug
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
> Attachments: ATLAS-885.patch
>
>
> For every 5 seconds following logs are added to 
> /var/logs/atlas/application.log, which increases the log file size 
> unnecessarily and makes the log difficult to use.
> {noformat}
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Process 
> identifier=hconnection-0x7878431f connecting to ZooKeeper 
> ensemble=localhost:2181 (RecoverableZooKeeper:120)
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Initiating client 
> connection, connectString=localhost:2181 sessionTimeout=9 
> watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@78b60e6a 
> (ZooKeeper:438)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Opening socket 
> connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt 
> to authenticate using SASL (unknown error) (ClientCnxn:1019)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Socket connection 
> established to localhost.localdomain/127.0.0.1:2181, initiating session 
> (ClientCnxn:864)
> 2016-05-17 22:01:18,531 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Session 
> establishment complete on server localhost.localdomain/127.0.0.1:2181, 
> sessionid = 0x154c1cf558e18b9, negotiated timeout = 4 (ClientCnxn:1279)
> 2016-05-17 22:01:18,539 INFO - [pool-5-thread-1:] ~ Closing zookeeper 
> sessionid=0x154c1cf558e18b9 (ConnectionManager$HConnectionImplementation:1684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1:] ~ Session: 
> 0x154c1cf558e18b9 closed (ZooKeeper:684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1-EventThread:] ~ EventThread 
> shut down (ClientCnxn:524)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-885) Excessive periodic logs to application.log

[Madhan Neethiraj (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321704#comment-15321704
 ] 

Madhan Neethiraj commented on ATLAS-885:


These logs were generated from Titan's interaction with HBase. 
HBaseStoreManager.getLocalKeyPartition() instantiates HTable object {{table = 
new HTable(hconf, tableName);}}. Per [~enis] (HBase committer), this is an 
expensive operation involving creation of TCP/IP connections (to zookeeper and 
meta server) and a large number of threads. He suggested to get the table from 
the connection object, instead of creating HTable directly. Verified that using 
the suggested approach stopped the excessive logs. I will attach the patch 
shortly.

Also, the logic used to determine if the storage is local or not, in 
HBaseStoreManager.getDeployment() looks incorrect. Even if one of the regions 
of the table exist in the localhost, this method returns the deployment as 
'local'. I am not sure how Titan uses the value returned from 
HBaseStoreManager.getDeployment() and what the impact of wrongly returning as 
local. In the attache patch, HBaseStoreManager.getDeployment() has been updated 
to return 'remote' - which should be the case for most production deployments. 
If this causes any significant performance overhead in single-node deployments, 
we can revise this logic.

These updates are in Titan implementation and we should have Titan community to 
review/consider this fix for appropriate versions.

> Excessive periodic logs to application.log
> --
>
> Key: ATLAS-885
> URL: https://issues.apache.org/jira/browse/ATLAS-885
> Project: Atlas
>  Issue Type: Bug
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>
> For every 5 seconds following logs are added to 
> /var/logs/atlas/application.log, which increases the log file size 
> unnecessarily and makes the log difficult to use.
> {noformat}
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Process 
> identifier=hconnection-0x7878431f connecting to ZooKeeper 
> ensemble=localhost:2181 (RecoverableZooKeeper:120)
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Initiating client 
> connection, connectString=localhost:2181 sessionTimeout=9 
> watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@78b60e6a 
> (ZooKeeper:438)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Opening socket 
> connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt 
> to authenticate using SASL (unknown error) (ClientCnxn:1019)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Socket connection 
> established to localhost.localdomain/127.0.0.1:2181, initiating session 
> (ClientCnxn:864)
> 2016-05-17 22:01:18,531 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Session 
> establishment complete on server localhost.localdomain/127.0.0.1:2181, 
> sessionid = 0x154c1cf558e18b9, negotiated timeout = 4 (ClientCnxn:1279)
> 2016-05-17 22:01:18,539 INFO - [pool-5-thread-1:] ~ Closing zookeeper 
> sessionid=0x154c1cf558e18b9 (ConnectionManager$HConnectionImplementation:1684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1:] ~ Session: 
> 0x154c1cf558e18b9 closed (ZooKeeper:684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1-EventThread:] ~ EventThread 
> shut down (ClientCnxn:524)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-885) optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 seconds

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-885:
---
Summary: optimize HBaseStoreManager to avoid expensive HTable instantiation 
every 5 seconds  (was: Excessive periodic logs to application.log)

> optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 
> seconds
> --
>
> Key: ATLAS-885
> URL: https://issues.apache.org/jira/browse/ATLAS-885
> Project: Atlas
>  Issue Type: Bug
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>
> For every 5 seconds following logs are added to 
> /var/logs/atlas/application.log, which increases the log file size 
> unnecessarily and makes the log difficult to use.
> {noformat}
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Process 
> identifier=hconnection-0x7878431f connecting to ZooKeeper 
> ensemble=localhost:2181 (RecoverableZooKeeper:120)
> 2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Initiating client 
> connection, connectString=localhost:2181 sessionTimeout=9 
> watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@78b60e6a 
> (ZooKeeper:438)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Opening socket 
> connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt 
> to authenticate using SASL (unknown error) (ClientCnxn:1019)
> 2016-05-17 22:01:18,529 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Socket connection 
> established to localhost.localdomain/127.0.0.1:2181, initiating session 
> (ClientCnxn:864)
> 2016-05-17 22:01:18,531 INFO - 
> pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Session 
> establishment complete on server localhost.localdomain/127.0.0.1:2181, 
> sessionid = 0x154c1cf558e18b9, negotiated timeout = 4 (ClientCnxn:1279)
> 2016-05-17 22:01:18,539 INFO - [pool-5-thread-1:] ~ Closing zookeeper 
> sessionid=0x154c1cf558e18b9 (ConnectionManager$HConnectionImplementation:1684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1:] ~ Session: 
> 0x154c1cf558e18b9 closed (ZooKeeper:684)
> 2016-05-17 22:01:18,540 INFO - [pool-5-thread-1-EventThread:] ~ EventThread 
> shut down (ClientCnxn:524)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-885) Excessive periodic logs to application.log

[Madhan Neethiraj (JIRA)]

Madhan Neethiraj created ATLAS-885:
--

 Summary: Excessive periodic logs to application.log
 Key: ATLAS-885
 URL: https://issues.apache.org/jira/browse/ATLAS-885
 Project: Atlas
  Issue Type: Bug
Reporter: Madhan Neethiraj
Assignee: Madhan Neethiraj


For every 5 seconds following logs are added to 
/var/logs/atlas/application.log, which increases the log file size 
unnecessarily and makes the log difficult to use.

{noformat}
2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Process 
identifier=hconnection-0x7878431f connecting to ZooKeeper 
ensemble=localhost:2181 (RecoverableZooKeeper:120)
2016-05-17 22:01:18,528 INFO - [pool-5-thread-1:] ~ Initiating client 
connection, connectString=localhost:2181 sessionTimeout=9 
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@78b60e6a 
(ZooKeeper:438)
2016-05-17 22:01:18,529 INFO - 
pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Opening socket 
connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to 
authenticate using SASL (unknown error) (ClientCnxn:1019)
2016-05-17 22:01:18,529 INFO - 
pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Socket connection 
established to localhost.localdomain/127.0.0.1:2181, initiating session 
(ClientCnxn:864)
2016-05-17 22:01:18,531 INFO - 
pool-5-thread-1-SendThread(localhost.localdomain:2181): ~ Session establishment 
complete on server localhost.localdomain/127.0.0.1:2181, sessionid = 
0x154c1cf558e18b9, negotiated timeout = 4 (ClientCnxn:1279)
2016-05-17 22:01:18,539 INFO - [pool-5-thread-1:] ~ Closing zookeeper 
sessionid=0x154c1cf558e18b9 (ConnectionManager$HConnectionImplementation:1684)
2016-05-17 22:01:18,540 INFO - [pool-5-thread-1:] ~ Session: 0x154c1cf558e18b9 
closed (ZooKeeper:684)
2016-05-17 22:01:18,540 INFO - [pool-5-thread-1-EventThread:] ~ EventThread 
shut down (ClientCnxn:524)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-642) import-hive should create the lineage for external tables

[Suma Shivaprasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321498#comment-15321498
 ] 

Suma Shivaprasad commented on ATLAS-642:


Can we use the template for create table query  from the show create table 
function since I see that "temporary" is missed. Not sure what else could be 
missing here

> import-hive should create the lineage for external tables
> -
>
> Key: ATLAS-642
> URL: https://issues.apache.org/jira/browse/ATLAS-642
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>Assignee: Vimal Sharma
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-642.patch
>
>
> import-hive can also start showing lineage between HDFS paths and tables. 
> However the create table process should also register the exact query that 
> was  used to create the table. Show create table does that by reconstructing 
> the query. Need to figure out if we can potentially use that to create the 
> query



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-751) Add support for primary key constraint on class types

[Suma Shivaprasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321398#comment-15321398
 ] 

Suma Shivaprasad commented on ATLAS-751:


Changes  completed

1. Type system changes to add primary key constraints on class types and 
serialization. Had faced issues in type serialization during this which was 
fixed.
2. Entity deduplication based on primary key attributes which in turn support 
gremlin query on primitive, class and an additional post processing check to 
handle array references
3. Model changes in UT models and HiveHook, Storm for adding primary key 
constraints
4. Remove unique key attributes across the model - Hive Hook, UTs, ITs in 
webapp, Storm, HDFS
5. Add APIs for get, delete, update by primary key
6. Made changes in Hive Hook to use the new APIs for update and delete.
7. Added support for evaluating a  primary key expression attribute - to 
maintain backward incompatibility and return qualifiedName during entity 
searches and gets.  - This allows user to specify the display format of the 
attribute and the value is evaluated at runtime using the attributes of the 
instance mapped from the graph.
8. Added tests in GraphBackedDiscoveryTest to make sure reference searches work 
- like for eg: table.db.name etc with multiple 'and' clauses 

Pending changes

1. Debug and Resolve performance issue with the Gremlin query(dedup) during 
writes - Currently have fixed it by running the query through TitanGraphQuery 
which returns a list of vertex ids which are then passed as the set of start 
vertices to further Gremlin queries for class reference searches . However 
there could be issues if there are a lot of vertex ids returned through the 
first filters and  still need to test if its fine and whats the limit which 
would cause it to break. Else need to debug and fix the issue with slow running 
queries.
2. Hive Hook ITs - are failing since column entity asserts are failing in a lot 
of cases(need to debug further) and secondly there is a change needed in 
deduplication to handle class vertex reference searches ( The code for this is 
in place but yet to be tested)
3. Falcon hook and Sqoop hook - merge changes after review and make changes in 
ITs
4. Storm hook - yet to test changes in model
5. Add some pending UTs in PrimaryKeyDedupHandler
6. Add ITs for new rest APIs added - for get , update, delete by primary key
7. Review and code review fixes 

> Add support for primary key constraint on class types
> -
>
> Key: ATLAS-751
> URL: https://issues.apache.org/jira/browse/ATLAS-751
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-751.1.patch, ATLAS-751.1.patch
>
>
> Persisting the qualified Name for an entity has multiple issues
> 1. In case of soft deletes,consider the following scenario
> a. Table A -> insert overwrite -> Table B
> If the table A  and B are dropped and recreated again and the insert 
> overwrite query is rerun, then it should create another lineage process since 
> the tables are different here. Similarly for CTAS etc. However the same 
> process will get updated due to the current way of storing qualifiedName 
> 2. Storing qualified names inherently leads to a lot of updates during 
> operations renames for eg: if a table is renames, all its 
> columns.qualifiedName, sd.qualifiedName etc get updated. If this is done with 
> partitions, then the updates will definitely take a lot of time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-884) Process registration should call Entity update instead of create

[Suma Shivaprasad (JIRA)]

Suma Shivaprasad created ATLAS-884:
--

 Summary: Process registration should call Entity update instead of 
create
 Key: ATLAS-884
 URL: https://issues.apache.org/jira/browse/ATLAS-884
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.6-incubating, 0.7-incubating
Reporter: Suma Shivaprasad
Assignee: Suma Shivaprasad
 Fix For: 0.7-incubating


Process registration in hive hook currently calls Entity create . But should 
call update entity instead 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-642) import-hive should create the lineage for external tables

[Suma Shivaprasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321121#comment-15321121
 ] 

Suma Shivaprasad commented on ATLAS-642:


Also, creating a process Referenceable is already present in HiveHook. Can we 
reuse that and share that between HMSB and HiveHook

> import-hive should create the lineage for external tables
> -
>
> Key: ATLAS-642
> URL: https://issues.apache.org/jira/browse/ATLAS-642
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>Assignee: Vimal Sharma
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-642.patch
>
>
> import-hive can also start showing lineage between HDFS paths and tables. 
> However the create table process should also register the exact query that 
> was  used to create the table. Show create table does that by reconstructing 
> the query. Need to figure out if we can potentially use that to create the 
> query



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (ATLAS-642) import-hive should create the lineage for external tables

[Suma Shivaprasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320969#comment-15320969
 ] 

Suma Shivaprasad edited comment on ATLAS-642 at 6/8/16 5:18 PM:


 lineageProcess.set(AtlasClient.REFERENCEABLE_ATTRIBUTE_NAME, 
tableQualifiedName); - this should be set of processQualifiedName - pls refer 
HiveHook where theres a function for getting the qualifiedName for a process - 
getProcessQualifiedName. The same for getProcessReference(tableQualifiedName);


was (Author: suma.shivaprasad):
 lineageProcess.set(AtlasClient.REFERENCEABLE_ATTRIBUTE_NAME, 
tableQualifiedName); - this should be set of processQualifiedName - pls refer 
HiveHook where theres a function for getting the qualifiedName for a process - 
getProcessQualifiedName

> import-hive should create the lineage for external tables
> -
>
> Key: ATLAS-642
> URL: https://issues.apache.org/jira/browse/ATLAS-642
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>Assignee: Vimal Sharma
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-642.patch
>
>
> import-hive can also start showing lineage between HDFS paths and tables. 
> However the create table process should also register the exact query that 
> was  used to create the table. Show create table does that by reconstructing 
> the query. Need to figure out if we can potentially use that to create the 
> query



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-642) import-hive should create the lineage for external tables

[Suma Shivaprasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320969#comment-15320969
 ] 

Suma Shivaprasad commented on ATLAS-642:


 lineageProcess.set(AtlasClient.REFERENCEABLE_ATTRIBUTE_NAME, 
tableQualifiedName); - this should be set of processQualifiedName - pls refer 
HiveHook where theres a function for getting the qualifiedName for a process - 
getProcessQualifiedName

> import-hive should create the lineage for external tables
> -
>
> Key: ATLAS-642
> URL: https://issues.apache.org/jira/browse/ATLAS-642
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>Assignee: Vimal Sharma
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-642.patch
>
>
> import-hive can also start showing lineage between HDFS paths and tables. 
> However the create table process should also register the exact query that 
> was  used to create the table. Show create table does that by reconstructing 
> the query. Need to figure out if we can potentially use that to create the 
> query



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48408: ATLAS-749 Make EntityAuditRepository implementation configurable

[Jeff Hagelberg]

> On June 8, 2016, 2:08 p.m., Jeff Hagelberg wrote:
> > One minor comment - there is really no point in accumulating the audit 
> > repository events in memory in production. I could see it potentially using 
> > a lot of memory, since it is never cleaned up.  It may make sense to add a 
> > Noop implementation that can be used in production environments where hbase 
> > is unavailable.

One more comment - atlas_start.py currently fails if hbase is not selected as 
the repository.  My changes for ATLAS-749 included logic to tolerate something 
other than hbase being used as the repository.  I also added the delete handler 
property to atlas-application.properties, since it was missing.  You can see 
the changes I made here: https://reviews.apache.org/r/48349/


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48408/#review136641
---


On June 8, 2016, 11:41 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48408/
> ---
> 
> (Updated June 8, 2016, 11:41 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-749
> https://issues.apache.org/jira/browse/ATLAS-749
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Makes audit repository implementation configurable
> 2. Made in memory audit repo as default for berkley, es profile
> 3. Updated docs
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 131a488 
>   distro/src/conf/atlas-application.properties 1cdd424 
>   docs/src/site/twiki/Configuration.twiki 7150483 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 15201dd 
>   typesystem/src/main/resources/atlas-application.properties 3318ff3 
>   webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 
> 010fa2a 
>   
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java
>  08bb125 
>   webapp/src/test/java/org/apache/atlas/web/listeners/TestModule.java 9cb76d3 
> 
> Diff: https://reviews.apache.org/r/48408/diff/
> 
> 
> Testing
> ---
> 
> Existing tests passed, tested end to end
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated June 8, 2016, 2:17 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description (updated)
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   - performance : add index on __state property, fix resulting test failure - 
see GraphBackedSearchIndexer, GraphHelper.getVertexForProperty
   - performance - consolidate creation of built-in types into single call.  
See DefaultMetadataService.
   
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.  You can, for the most part, ignore 
the new files under graphdb\titan0.  There are a few minor changes that have 
been put in since ATLAS-693, but nothing very significant.  Just changes to 
implement a couple new methods in the abstraction layer.

Last rebase: 6/7/2016


Diffs
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
ba8e0e712769afad72f7f8254ec626a1d17136e8 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.jav

[jira] [Resolved] (ATLAS-871) Make audit repository implementation configurable

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg resolved ATLAS-871.
-
Resolution: Duplicate

> Make audit repository implementation configurable
> -
>
> Key: ATLAS-871
> URL: https://issues.apache.org/jira/browse/ATLAS-871
> Project: Atlas
>  Issue Type: Task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48408: ATLAS-749 Make EntityAuditRepository implementation configurable

[Jeff Hagelberg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48408/#review136641
---



One minor comment - there is really no point in accumulating the audit 
repository events in memory in production. I could see it potentially using a 
lot of memory, since it is never cleaned up.  It may make sense to add a Noop 
implementation that can be used in production environments where hbase is 
unavailable.

- Jeff Hagelberg


On June 8, 2016, 11:41 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48408/
> ---
> 
> (Updated June 8, 2016, 11:41 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-749
> https://issues.apache.org/jira/browse/ATLAS-749
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Makes audit repository implementation configurable
> 2. Made in memory audit repo as default for berkley, es profile
> 3. Updated docs
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 131a488 
>   distro/src/conf/atlas-application.properties 1cdd424 
>   docs/src/site/twiki/Configuration.twiki 7150483 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 15201dd 
>   typesystem/src/main/resources/atlas-application.properties 3318ff3 
>   webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 
> 010fa2a 
>   
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java
>  08bb125 
>   webapp/src/test/java/org/apache/atlas/web/listeners/TestModule.java 9cb76d3 
> 
> Diff: https://reviews.apache.org/r/48408/diff/
> 
> 
> Testing
> ---
> 
> Existing tests passed, tested end to end
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

[jira] [Commented] (ATLAS-871) Make audit repository implementation configurable

[Jeffrey Hagelberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320601#comment-15320601
 ] 

Jeffrey Hagelberg commented on ATLAS-871:
-

Ok, great.  The real difference I see is that in my implementation I added a 
Noop implementation, since there is really no point in accumulating the audit 
repository events in memory in production.  I could see it potentially using a 
lot of memory, since it is nver cleaned up.

> Make audit repository implementation configurable
> -
>
> Key: ATLAS-871
> URL: https://issues.apache.org/jira/browse/ATLAS-871
> Project: Atlas
>  Issue Type: Task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-877) CreateTime of an entity(for example hive table) changes for every update

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-877:
---
Assignee: Shwetha G S

> CreateTime of an entity(for example hive table) changes for every update
> 
>
> Key: ATLAS-877
> URL: https://issues.apache.org/jira/browse/ATLAS-877
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Shwetha G S
>Priority: Critical
> Fix For: 0.7-incubating
>
>
> Each time a hive table is updated(example column renamed), create time of 
> table is also updated.
> UI snapshot: https://monosnap.com/file/6NoT86CNpHgQmlyfBQTSUa1y3Ncc2Y
> hive table metadata snapshot from hive cli
> {noformat}
> 0: jdbc:hive2://localhost:1/default> describe formatted 
> column_changepgwcbgq1bd;
> OK
> +---+-+---+--+
> |   col_name|  
> data_type  |comment|
> +---+-+---+--+
> | # col_name| data_type   
> | comment   |
> |   | NULL
> | NULL  |
> | a3| int 
> |   |
> | b | int 
> |   |
> | c | int 
> |   |
> | d | int 
> |   |
> | e | int 
> |   |
> | f | int 
> |   |
> |   | NULL
> | NULL  |
> | # Detailed Table Information  | NULL
> | NULL  |
> | Database: | db2pgwcbgq1bd   
> | NULL  |
> | Owner:| apathan 
> | NULL  |
> | CreateTime:   | Wed Jun 08 12:41:28 IST 2016
> | NULL  |
> | LastAccessTime:   | UNKNOWN 
> | NULL  |
> | Protect Mode: | None
> | NULL  |
> | Retention:| 0   
> | NULL  |
> | Location: | 
> hdfs://localhost:9000/user/hive/warehouse/db2pgwcbgq1bd.db/column_changepgwcbgq1bd
>   | NULL  |
> | Table Type:   | MANAGED_TABLE   
> | NULL  |
> | Table Parameters: | NULL
> | NULL  |
> |   | COLUMN_STATS_ACCURATE   
> | false |
> |   | last_modified_by
> | apathan   |
> |   | last_modified_time  
> | 1465370834|
> |   | numFiles
> | 0 |
> |   | numRows  

[jira] [Updated] (ATLAS-878) UI: Not showing details of SD, DB and COLUMNS

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-878:
---
Assignee: Keval Bhatt

> UI: Not showing details of SD, DB and COLUMNS
> -
>
> Key: ATLAS-878
> URL: https://issues.apache.org/jira/browse/ATLAS-878
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Keval Bhatt
>Priority: Blocker
> Fix For: 0.7-incubating
>
>
> UI: Not showing details of SD, DB and COLUMNS
> UI snapshot: https://monosnap.com/file/TEv26M69LDwRctYgPKpyM2pmYLSl6R



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (ATLAS-880) Audit log: Tag association to hive column is not logged

[Ayub Khan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320568#comment-15320568
 ] 

Ayub Khan edited comment on ATLAS-880 at 6/8/16 1:41 PM:
-

[~shwethags] Yes, I was referring to association under table entity.


was (Author: ayubkhan):
[~shwethags] Yes, I mean association under table entity.

> Audit log: Tag association to hive column is not logged
> ---
>
> Key: ATLAS-880
> URL: https://issues.apache.org/jira/browse/ATLAS-880
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Shwetha G S
> Fix For: 0.7-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-880) Audit log: Tag association to hive column is not logged

[Ayub Khan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320568#comment-15320568
 ] 

Ayub Khan commented on ATLAS-880:
-

[~shwethags] Yes, I mean association under table entity.

> Audit log: Tag association to hive column is not logged
> ---
>
> Key: ATLAS-880
> URL: https://issues.apache.org/jira/browse/ATLAS-880
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Shwetha G S
> Fix For: 0.7-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-642) import-hive should create the lineage for external tables

[Vimal Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320469#comment-15320469
 ] 

Vimal Sharma commented on ATLAS-642:


We can create the lineage by extracting metadata from Hive table. We need to 
create a hive process with HDFS path and table as input and output respectively.

I have submit a patch with this change. We can revisit the task of getting the 
"SHOW CREATE TABLE" query for a table in another JIRA.

> import-hive should create the lineage for external tables
> -
>
> Key: ATLAS-642
> URL: https://issues.apache.org/jira/browse/ATLAS-642
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>Assignee: Vimal Sharma
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-642.patch
>
>
> import-hive can also start showing lineage between HDFS paths and tables. 
> However the create table process should also register the exact query that 
> was  used to create the table. Show create table does that by reconstructing 
> the query. Need to figure out if we can potentially use that to create the 
> query



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-642) import-hive should create the lineage for external tables

[Vimal Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vimal Sharma updated ATLAS-642:
---
Attachment: ATLAS-642.patch

> import-hive should create the lineage for external tables
> -
>
> Key: ATLAS-642
> URL: https://issues.apache.org/jira/browse/ATLAS-642
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>Assignee: Vimal Sharma
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-642.patch
>
>
> import-hive can also start showing lineage between HDFS paths and tables. 
> However the create table process should also register the exact query that 
> was  used to create the table. Show create table does that by reconstructing 
> the query. Need to figure out if we can potentially use that to create the 
> query



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (ATLAS-853) User's name to be mentioned in the top user drop down

[Saqeeb Shaikh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saqeeb Shaikh updated ATLAS-853:

Comment: was deleted

(was: Fix for improper user name in kerberized setup. Changed the method to 
getName() from getPrincipal(), to obtain the user name from the Security 
Context.)

> User's name to be mentioned in the top user drop down
> -
>
> Key: ATLAS-853
> URL: https://issues.apache.org/jira/browse/ATLAS-853
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 0.7-incubating
>Reporter: Saqeeb Shaikh
>Assignee: Saqeeb Shaikh
>  Labels: patch
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-853.1.patch, ATLAS-853.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-853) User's name to be mentioned in the top user drop down

[Saqeeb Shaikh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saqeeb Shaikh updated ATLAS-853:

Attachment: ATLAS-853.1.patch

Fix for improper user name in kerberized setup. Changed the method to getName() 
from getPrincipal(), to obtain the user name from the Security Context.

{code:title=Kerberos curl test |borderStyle=solid}
curl -k -v  --negotiate -u :  
http://mp-atls-495-1.openstacklocal:21000/api/atlas/admin/session

< HTTP/1.1 200 OK
< Date: Wed, 08 Jun 2016 11:50:23 GMT
< Set-Cookie: JSESSIONID=1j5vc9ikrffu18tsza3hepgef;Path=/
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: 
hadoop.auth="u=atlas&p=atlas/mp-atls-495-1.openstacklo...@example.com&t=kerberos&e=1465422623250&s=m8QVW9W2hGoOuHPmx1qExJbGwBs=";
 Version=1; Path=/; Expires=Wed, 08-Jun-2016 21:50:23 GMT; HttpOnly
< Content-Type: application/json; charset=UTF-8
< Transfer-Encoding: chunked
< Server: Jetty(9.2.12.v20150709)
< 
* Connection #0 to host mp-atls-495-1.openstacklocal left intact
* Closing connection #0

{"userName":"atlas","groups":["hadoop"]}
{code}



> User's name to be mentioned in the top user drop down
> -
>
> Key: ATLAS-853
> URL: https://issues.apache.org/jira/browse/ATLAS-853
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 0.7-incubating
>Reporter: Saqeeb Shaikh
>Assignee: Saqeeb Shaikh
>  Labels: patch
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-853.1.patch, ATLAS-853.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-853) User's name to be mentioned in the top user drop down

[Saqeeb Shaikh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saqeeb Shaikh updated ATLAS-853:

Attachment: (was: ATLAS-853.patch)

> User's name to be mentioned in the top user drop down
> -
>
> Key: ATLAS-853
> URL: https://issues.apache.org/jira/browse/ATLAS-853
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 0.7-incubating
>Reporter: Saqeeb Shaikh
>Assignee: Saqeeb Shaikh
>  Labels: patch
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-853.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-853) User's name to be mentioned in the top user drop down

[Saqeeb Shaikh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saqeeb Shaikh updated ATLAS-853:

Attachment: ATLAS-853.patch

Fix for improper user name in kerberized setup. Changed the method to getName() 
from getPrincipal(), to obtain the user name from the Security Context.

> User's name to be mentioned in the top user drop down
> -
>
> Key: ATLAS-853
> URL: https://issues.apache.org/jira/browse/ATLAS-853
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: 0.7-incubating
>Reporter: Saqeeb Shaikh
>Assignee: Saqeeb Shaikh
>  Labels: patch
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-853.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48339: ATLAS-853 User's name to be mentioned in the top user drop down

[Saqeeb Shaikh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48339/
---

(Updated June 8, 2016, 11:55 a.m.)


Review request for atlas, Gautam Borad, Madhan Neethiraj, Mehul Parikh, Shwetha 
GS, and Hemanth Yamijala.


Changes
---

Fix for improper user name in kerberized setup. Changed the method to getName() 
from getPrincipal(), to obtain the user name from the Security Context.


Bugs: ATLAS-853
https://issues.apache.org/jira/browse/ATLAS-853


Repository: atlas


Description
---

Added new API /api/atlas/admin/session to obtain logged in user information. 
This is a part of Admin API.


Diffs (updated)
-

  
authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasAuthorizationUtils.java
 48b536f 
  dashboardv2/public/js/templates/business_catalog/BusinessCatalogHeader.html 
006fe0b 
  dashboardv2/public/js/templates/site/header.html 8c22e5a 
  dashboardv2/public/js/utils/CommonViewFunction.js d2de0dc 
  dashboardv2/public/js/utils/Globals.js 078881e 
  dashboardv2/public/js/views/business_catalog/BusinessCatalogHeader.js aad4bab 
  dashboardv2/public/js/views/site/Header.js cf97a75 
  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthorizationFilter.java 
d87120c 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
36b7607 

Diff: https://reviews.apache.org/r/48339/diff/


Testing
---

mvn install and package

The API call to test the result JSON :: /api/atlas/admin/session

Testing result :: {"userName":"admin","groups":["ADMIN"]}


Thanks,

Saqeeb Shaikh

Build failed in Jenkins: apache-atlas-nightly #306

[Apache Jenkins Server]

See 

Changes:

[hyamijala] ATLAS-870 Add search feature while associating Tags / Terms with 
entity.

--
[...truncated 5923 lines...]
[INFO] Using incremental compilation
[INFO] Compiling 56 Java sources to 

[WARNING] Note: Some input files use or override a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[WARNING] Note: Some input files use unchecked or unsafe operations.
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ atlas-webapp 
---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 56 source files to 

[WARNING] 
:
 Some input files use or override a deprecated API.
[WARNING] 
:
 Recompile with -Xlint:deprecation for details.
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ 
atlas-webapp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- scala-maven-plugin:3.2.0:testCompile (scala-test-compile-first) @ 
atlas-webapp ---
[WARNING] Zinc server is not available at port 3030 - reverting to normal 
incremental compile
[INFO] Using incremental compilation
[INFO] Compiling 41 Java sources to 

[WARNING] Note: Some input files use unchecked or unsafe operations.
[WARNING] Note: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
atlas-webapp ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 41 source files to 

[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-war-plugin:2.6:war (default-war) @ atlas-webapp ---
[INFO] Packaging webapp
[INFO] Assembling webapp [atlas-webapp] in 
[
[INFO] Processing overlay [ id org.apache.atlas:atlas-dashboardv2]
[INFO] Processing war project
[INFO] Copying webapp resources 
[
[INFO] Building jar: 

[INFO] Webapp assembled in [1822 msecs]
[INFO] Building war: 

[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
atlas-webapp ---
[INFO] 
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ atlas-webapp ---
[WARNING] The parameter forkMode is deprecated since version 2.14. Use 
forkCount and reuseForks instead.
[INFO] Surefire report directory: 

[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[WARNING] file.encoding cannot be set as system property, use 
-Dfile.encoding=... instead

---
 T E S T S
---
Running org.apache.atlas.web.service.ActiveInstanceElectorServiceTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.256 sec - in 
org.apache.atlas.web.service.ActiveInstanceElectorServiceTest
Running org.apache.atlas.web.service.CuratorFactoryTest
Tests run: 3, Failures: 0, Errors:

Review Request 48408: ATLAS-749 Make EntityAuditRepository implementation configurable

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48408/
---

Review request for atlas.


Bugs: ATLAS-749
https://issues.apache.org/jira/browse/ATLAS-749


Repository: atlas


Description
---

1. Makes audit repository implementation configurable
2. Made in memory audit repo as default for berkley, es profile
3. Updated docs


Diffs
-

  distro/pom.xml 131a488 
  distro/src/conf/atlas-application.properties 1cdd424 
  docs/src/site/twiki/Configuration.twiki 7150483 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
15201dd 
  typesystem/src/main/resources/atlas-application.properties 3318ff3 
  webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 
010fa2a 
  
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 
08bb125 
  webapp/src/test/java/org/apache/atlas/web/listeners/TestModule.java 9cb76d3 

Diff: https://reviews.apache.org/r/48408/diff/


Testing
---

Existing tests passed, tested end to end


Thanks,

Shwetha GS

[jira] [Updated] (ATLAS-749) Make EntityAuditRepository implementation configurable

[Shwetha G S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S updated ATLAS-749:
--
Attachment: ATLAS-749.patch

> Make EntityAuditRepository implementation configurable
> --
>
> Key: ATLAS-749
> URL: https://issues.apache.org/jira/browse/ATLAS-749
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Shwetha G S
>Assignee: Shwetha G S
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-749.patch
>
>
> EntityAuditRepository is bound to HBaseBasedAuditRepository in 
> RepositoryMetadataModule, which essentially forces hbase setup. Make it 
> configurable using atlas-application.properties on the lines of DeleteHandler 
> configuration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-880) Audit log: Tag association to hive column is not logged

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320396#comment-15320396
 ] 

Shwetha G S commented on ATLAS-880:
---

[~ayubkhan], associating tag to column is logged under the column entity. I 
verified that it works. Did you mean that this association is not logged under 
table entity?

> Audit log: Tag association to hive column is not logged
> ---
>
> Key: ATLAS-880
> URL: https://issues.apache.org/jira/browse/ATLAS-880
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Shwetha G S
> Fix For: 0.7-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-883) Two HTTP 404 calls to /lineage are made while viewing entity

[Vishwanath Savakar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwanath Savakar updated ATLAS-883:
-
Attachment: screenshot-2.png

> Two HTTP 404 calls to /lineage are made while viewing entity
> 
>
> Key: ATLAS-883
> URL: https://issues.apache.org/jira/browse/ATLAS-883
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Vishwanath Savakar
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> Searched for the entity and after clicking on the displayed results, observed 
> two calls leading to HTTP 404. Screenshot attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-883) Two HTTP 404 calls to /lineage are made while viewing entity

[Vishwanath Savakar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwanath Savakar updated ATLAS-883:
-
Attachment: screenshot-1.png

> Two HTTP 404 calls to /lineage are made while viewing entity
> 
>
> Key: ATLAS-883
> URL: https://issues.apache.org/jira/browse/ATLAS-883
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Vishwanath Savakar
> Attachments: screenshot-1.png
>
>
> Searched for the entity and after clicking on the displayed results, observed 
> two calls leading to HTTP 404. Screenshot attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-883) Two HTTP 404 calls to /lineage are made while viewing entity

[Vishwanath Savakar (JIRA)]

Vishwanath Savakar created ATLAS-883:


 Summary: Two HTTP 404 calls to /lineage are made while viewing 
entity
 Key: ATLAS-883
 URL: https://issues.apache.org/jira/browse/ATLAS-883
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.7-incubating
Reporter: Vishwanath Savakar


Searched for the entity and after clicking on the displayed results, observed 
two calls leading to HTTP 404. Screenshot attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-882) with multiple taxonomies, UI does not allow associating terms with entities. "Add Terms" drop down shows all the taxonomies

[Ayub Khan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayub Khan updated ATLAS-882:

Attachment: Screen Shot 2016-06-08 at 4.20.45 PM.png

> with multiple taxonomies, UI does not allow associating terms with entities. 
> "Add Terms" drop down shows all the taxonomies
> ---
>
> Key: ATLAS-882
> URL: https://issues.apache.org/jira/browse/ATLAS-882
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Priority: Blocker
> Fix For: 0.7-incubating
>
> Attachments: Screen Shot 2016-06-08 at 4.20.45 PM.png
>
>
> with multiple taxonomies, UI does not allow associating terms with entities. 
> "Add Terms" drop down shows all the taxonomies
> Attaching UI snapshot.
> Create multiple taxonomies with terms and try associating terms to entities, 
> UI does not allow it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-882) with multiple taxonomies, UI does not allow associating terms with entities. "Add Terms" drop down shows all the taxonomies

[Ayub Khan (JIRA)]

Ayub Khan created ATLAS-882:
---

 Summary: with multiple taxonomies, UI does not allow associating 
terms with entities. "Add Terms" drop down shows all the taxonomies
 Key: ATLAS-882
 URL: https://issues.apache.org/jira/browse/ATLAS-882
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.7-incubating
Reporter: Ayub Khan
Priority: Blocker
 Fix For: 0.7-incubating


with multiple taxonomies, UI does not allow associating terms with entities. 
"Add Terms" drop down shows all the taxonomies

Attaching UI snapshot.

Create multiple taxonomies with terms and try associating terms to entities, UI 
does not allow it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (ATLAS-876) No lineage for a re-created ctas table

[Shwetha G S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S reassigned ATLAS-876:
-

Assignee: Shwetha G S

> No lineage for a re-created ctas table
> --
>
> Key: ATLAS-876
> URL: https://issues.apache.org/jira/browse/ATLAS-876
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Shwetha G S
>Priority: Critical
> Fix For: 0.7-incubating
>
>
> No lineage for a re-created ctas table
> Steps to reproduce:
> create table account (name String, balance Int);
> create table account_ctas as select * from account;
> drop table account_ctas;
> create table account_ctas as select * from account;
> Empty responses from server for recreated ctas
> {noformat}
> curl 
> 'http://localhost:21000/api/atlas/lineage/ebd6a5a9-8453-4dbd-b8d9-1575a464ebc3/inputs/graph'
>  -H 'Cookie: JSESSIONID=12rn8aybkawa5ewiqdb2hv663' -H 'Accept-Encoding: gzip, 
> deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: 
> Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, 
> like Gecko) Chrome/50.0.2661.102 Safari/537.36' -H 'Accept: application/json, 
> text/javascript, */*; q=0.01' -H 'Referer: http://localhost:21000/index.html' 
> -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' 
> --compressed | python -m json.tool
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
> 100   2370   2370 0281  0 --:--:-- --:--:-- --:--:--   281
> {
> "requestId": "qtp1211076369-876 - ad29b29b-7faf-44f2-a2bd-8bbb00f07120",
> "results": {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
> "typeName": "__tempQueryResultStruct135",
> "values": {
> "edges": {},
> "vertices": {}
> }
> }
> }
> curl 
> 'http://localhost:21000/api/atlas/lineage/ebd6a5a9-8453-4dbd-b8d9-1575a464ebc3/outputs/graph'
>  -H 'Cookie: JSESSIONID=12rn8aybkawa5ewiqdb2hv663' -H 'Accept-Encoding: gzip, 
> deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: 
> Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, 
> like Gecko) Chrome/50.0.2661.102 Safari/537.36' -H 'Accept: application/json, 
> text/javascript, */*; q=0.01' -H 'Referer: http://localhost:21000/index.html' 
> -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' 
> --compressed | python -m json.tool
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
> 100   2380   2380 0271  0 --:--:-- --:--:-- --:--:--   271
> {
> "requestId": "qtp1211076369-1104 - ee65b8d3-0cf9-4477-a644-d93aab6b625b",
> "results": {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
> "typeName": "__tempQueryResultStruct141",
> "values": {
> "edges": {},
> "vertices": {}
> }
> }
> }
> {noformat}
> UI snapshot: https://monosnap.com/file/3XcCcffPjrayjLaIByWcjBHU4k2pgV



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-871) Make audit repository implementation configurable

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320337#comment-15320337
 ] 

Shwetha G S commented on ATLAS-871:
---

[~jnhagelb], I am working on this in ATLAS-749

> Make audit repository implementation configurable
> -
>
> Key: ATLAS-871
> URL: https://issues.apache.org/jira/browse/ATLAS-871
> Project: Atlas
>  Issue Type: Task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (ATLAS-749) Make EntityAuditRepository implementation configurable

[Shwetha G S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S reassigned ATLAS-749:
-

Assignee: Shwetha G S

> Make EntityAuditRepository implementation configurable
> --
>
> Key: ATLAS-749
> URL: https://issues.apache.org/jira/browse/ATLAS-749
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Shwetha G S
>Assignee: Shwetha G S
> Fix For: 0.7-incubating
>
>
> EntityAuditRepository is bound to HBaseBasedAuditRepository in 
> RepositoryMetadataModule, which essentially forces hbase setup. Make it 
> configurable using atlas-application.properties on the lines of DeleteHandler 
> configuration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-881) UI: not displaying multiple taxonomies

[Ayub Khan (JIRA)]

Ayub Khan created ATLAS-881:
---

 Summary: UI: not displaying multiple taxonomies
 Key: ATLAS-881
 URL: https://issues.apache.org/jira/browse/ATLAS-881
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.7-incubating
Reporter: Ayub Khan
Priority: Critical
 Fix For: 0.7-incubating


UI: not displaying multiple taxonomies


API response returns multiple taxonomies
{noformat}
curl 'http://localhost:21000/api/atlas/v1/taxonomies/' -H 'Cookie: 
JSESSIONID=19eplhd4dz8mueh7466qcbjwf' -H 'Accept-Encoding: gzip, deflate, sdch' 
-H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; 
Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/50.0.2661.102 Safari/537.36' -H 'Accept: application/json, 
text/javascript, */*; q=0.01' -H 'Referer: http://localhost:21000/index.html' 
-H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' --compressed 
| python -m json.tool
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
100   3310   3310 0   4096  0 --:--:-- --:--:-- --:--:--  4137
[
{
"href": "http://localhost:21000/api/atlas/v1/taxonomies/empty";,
"name": "empty"
},
{
"href": "http://localhost:21000/api/atlas/v1/taxonomies/tax1";,
"name": "tax1"
},
{
"href": "http://localhost:21000/api/atlas/v1/taxonomies/default";,
"name": "default"
}
]
{noformat}

UI snapshot: https://monosnap.com/file/1M6dOB8D08IPMqkqj3LsCr8VYvXWJG



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-515) Ability to initialize Kafka topics with more than 1 replica

[Hemanth Yamijala (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320301#comment-15320301
 ] 

Hemanth Yamijala commented on ATLAS-515:


Just picking this thread up once more. From some analysis, it appears that the 
secure Kafka restrictions do impose a huge inconvenience in how this would 
work. An option, therefore, is to convert this to a utility (rather than doing 
it automatically during regular runtime work of the hooks or server). This 
utility can either be invoked as the right user, or potentially the 
authentication can happen in line by the utility itself when launched with the 
right parameters (for e.g. it can do a kinit programmatically as a service 
user). 

The reason and way to run this utility should be documented. Running it can be 
automated via deployment tools like Ambari. I will convert the patch to be like 
this, by:

* Removing calling the AtlasTopicCreator from initialization in hooks, server 
side
* Creating a shell script that launches this.

I still have a few questions about how the login should be done automatically 
for secure Kafka deployments. Will update once I find out.

> Ability to initialize Kafka topics with more than 1 replica
> ---
>
> Key: ATLAS-515
> URL: https://issues.apache.org/jira/browse/ATLAS-515
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-515-poc.patch, ATLAS-515.patch
>
>
> Atlas uses two Kafka topics - ATLAS_HOOK and ATLAS_ENTITIES for communicating 
> with its upstream and downstream dependencies. If not explicitly created, 
> these topics are setup with a single replica. From a HA perspective, this 
> becomes a problem. This JIRA is to request for a mechanism for creating 
> topics with more than 1 replica. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-880) Audit log: Tag association to hive column is not logged

[Ayub Khan (JIRA)]

Ayub Khan created ATLAS-880:
---

 Summary: Audit log: Tag association to hive column is not logged
 Key: ATLAS-880
 URL: https://issues.apache.org/jira/browse/ATLAS-880
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.7-incubating
Reporter: Ayub Khan
Assignee: Shwetha G S
 Fix For: 0.7-incubating






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-879) Business catalog: Terms are audited as tags.

[Ayub Khan (JIRA)]

Ayub Khan created ATLAS-879:
---

 Summary: Business catalog: Terms are audited as tags.
 Key: ATLAS-879
 URL: https://issues.apache.org/jira/browse/ATLAS-879
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.7-incubating
Reporter: Ayub Khan
 Fix For: 0.7-incubating


Is there a need to differentiate terms from tags from an audit perspective?

>From a implementation point of view, terms are modelled as tags. But from an 
>end user perspective, terms are different from tags. In auditing, terms audit 
>logs are displayed as tag audit.

UI Snapshot: https://monosnap.com/file/560YS5CRHRioAoRbdDpRSP7Alm8o2B



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: apache-atlas-nightly #305

[Apache Jenkins Server]

See 

Changes:

[hyamijala] ATLAS-820 Kerberized env: Authentication failing (nixonrodrigues via

[hyamijala] ATLAS-865 Edit description functionality for Tags (kevalbhatt18 via

--
[...truncated 2376 lines...]
:239:
 Line matches the illegal pattern 'Trailing whitespace'.
:243:
 Line matches the illegal pattern 'Trailing whitespace'.
:248:
 Line matches the illegal pattern 'Trailing whitespace'.
:28:
 Line matches the illegal pattern 'Trailing whitespace'.
:33:
 First sentence should end with a period.
:45:
 First sentence should end with a period.
:47:
 Line matches the illegal pattern 'Trailing whitespace'.
:52:
 Line matches the illegal pattern 'Trailing whitespace'.
:60:
 Line matches the illegal pattern 'Trailing whitespace'.
:64:
 Line matches the illegal pattern 'Trailing whitespace'.
:65:
 Line matches the illegal pattern 'Trailing whitespace'.
:66:
 Line matches the illegal pattern 'Trailing whitespace'.
:67:
 First sentence should end with a period.
:70:
 Line is longer than 120 characters.
:72:
 Line matches the illegal pattern 'Trailing whitespace'.
:73:
 Line matches the illegal pattern 'Trailing whitespace'.
:25:
 Line matches the illegal pattern 'Trailing whitespace'.
:29:36:
 ',' is not followed by whitespace.
:33:
 Line matches the illegal pattern 'Trailing whitespace'.
:36:23:
 ',' is not followed by whitespace.
:38:
 Line matches the illegal pattern 'Trailing whitespace'.
:40:
 Line matches the illegal pattern 'Trailing whitespace'.
:44:31:
 ',' is not followed by whitespace.
:45:
 Line matches the illegal pattern 'Trailing whitespace'.

[jira] [Created] (ATLAS-878) UI: Not showing details of SD, DB and COLUMNS

[Ayub Khan (JIRA)]

Ayub Khan created ATLAS-878:
---

 Summary: UI: Not showing details of SD, DB and COLUMNS
 Key: ATLAS-878
 URL: https://issues.apache.org/jira/browse/ATLAS-878
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.7-incubating
Reporter: Ayub Khan
Priority: Blocker
 Fix For: 0.7-incubating


UI: Not showing details of SD, DB and COLUMNS

UI snapshot: https://monosnap.com/file/TEv26M69LDwRctYgPKpyM2pmYLSl6R



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-877) CreateTime of an entity(for example hive table) changes for every update

[Ayub Khan (JIRA)]

Ayub Khan created ATLAS-877:
---

 Summary: CreateTime of an entity(for example hive table) changes 
for every update
 Key: ATLAS-877
 URL: https://issues.apache.org/jira/browse/ATLAS-877
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.7-incubating
Reporter: Ayub Khan
Priority: Critical
 Fix For: 0.7-incubating


Each time a hive table is updated(example column renamed), create time of table 
is also updated.

UI snapshot: https://monosnap.com/file/6NoT86CNpHgQmlyfBQTSUa1y3Ncc2Y

hive table metadata snapshot from hive cli
{noformat}
0: jdbc:hive2://localhost:1/default> describe formatted 
column_changepgwcbgq1bd;
OK
+---+-+---+--+
|   col_name|  
data_type  |comment|
+---+-+---+--+
| # col_name| data_type 
  | comment   |
|   | NULL  
  | NULL  |
| a3| int   
  |   |
| b | int   
  |   |
| c | int   
  |   |
| d | int   
  |   |
| e | int   
  |   |
| f | int   
  |   |
|   | NULL  
  | NULL  |
| # Detailed Table Information  | NULL  
  | NULL  |
| Database: | db2pgwcbgq1bd 
  | NULL  |
| Owner:| apathan   
  | NULL  |
| CreateTime:   | Wed Jun 08 12:41:28 IST 2016  
  | NULL  |
| LastAccessTime:   | UNKNOWN   
  | NULL  |
| Protect Mode: | None  
  | NULL  |
| Retention:| 0 
  | NULL  |
| Location: | 
hdfs://localhost:9000/user/hive/warehouse/db2pgwcbgq1bd.db/column_changepgwcbgq1bd
  | NULL  |
| Table Type:   | MANAGED_TABLE 
  | NULL  |
| Table Parameters: | NULL  
  | NULL  |
|   | COLUMN_STATS_ACCURATE 
  | false |
|   | last_modified_by  
  | apathan   |
|   | last_modified_time
  | 1465370834|
|   | numFiles  
  | 0 |
|   | numRows   
  | -1|
|   | rawDataSize   
  | -1|
|   | totalSize 
  | 0 |
|  

Re: Review Request 48339: ATLAS-853 User's name to be mentioned in the top user drop down

[Madhan Neethiraj]

Saqeeb,

 

Login using Kerberos shows the following text in the username field. Can you 
please review and update?

 

 

 

Thanks,

Madhan

 

 

From: Saqeeb Shaikh  on behalf of Saqeeb Shaikh 

Reply-To: Saqeeb Shaikh 
Date: Tuesday, June 7, 2016 at 5:35 AM
To: Hemanth Yamijala , Madhan Neethiraj 
, Mehul Parikh , Gautam Borad 
, Shwetha Shivalingamurthy 
Cc: atlas , Saqeeb Shaikh 

Subject: Review Request 48339: ATLAS-853 User's name to be mentioned in the top 
user drop down

 

This is an automatically generated e-mail. To reply, visit: 
https://reviews.apache.org/r/48339/ 

 

Review request for atlas, Gautam Borad, Madhan Neethiraj, Mehul Parikh, Shwetha 
GS, and Hemanth Yamijala.
By Saqeeb Shaikh.
Bugs: ATLAS-853 
Repository: atlas 
Description 
Added new API /api/atlas/admin/session to obtain logged in user information. 
This is a part of Admin API.
Testing 
mvn install and package
 
The API call to test the result JSON :: /api/atlas/admin/session
 
Testing result :: {"userName":"admin","groups":["ADMIN"]}
Diffs 
authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasAuthorizationUtils.java
 (48b536f) 
dashboardv2/public/js/templates/business_catalog/BusinessCatalogHeader.html 
(006fe0b) 
dashboardv2/public/js/templates/site/header.html (8c22e5a) 
dashboardv2/public/js/utils/CommonViewFunction.js (3baeb34) 
dashboardv2/public/js/utils/Globals.js (078881e) 
dashboardv2/public/js/views/business_catalog/BusinessCatalogHeader.js (aad4bab) 
dashboardv2/public/js/views/site/Header.js (cf97a75) 
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
(36b7607) 
View Diff