[jira] [Comment Edited] (ATLAS-529) support drop database with restrict(no cascades)
[ https://issues.apache.org/jira/browse/ATLAS-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231186#comment-15231186 ] Suma Shivaprasad edited comment on ATLAS-529 at 4/7/16 10:03 PM: - Attaching rebased patch on top of ATLAS-528. Needs to be committed after ATLAS-528 was (Author: suma.shivaprasad): Attaching rebased patch on top of ATLAS-528 > support drop database with restrict(no cascades) > > > Key: ATLAS-529 > URL: https://issues.apache.org/jira/browse/ATLAS-529 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-529.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230983#comment-15230983 ] Suma Shivaprasad edited comment on ATLAS-528 at 4/7/16 8:31 PM: [~yhemanth] Thanks for the review. The handling of truncate table will create a single process with no inputs and a single table as output. We should validate if this type of lineage view is OK for a end user. I agree though that capturing the information is important. --> Raised ATLAS-653 to track this. On the same note, is the query string captured correctly for the process. Can we enhance the truncate test to validate this? --> HiveHookIT.validateProcess already check this by querying with the same exact query If a table / view being dropped is not present in Atlas, this would generate a 404 and we would loose the information. Maybe we can file a bug and track this for later? --> Raised ATLAS-652 to track this. We should also test what happens if a user executes a drop table if exists non_existent_table - do we get called? --> I observed that the hook is being called and the hook is ignoring this since there are no outputs for this query. I dont think we should capture these kind of queries . Please let me know if you feel otherwise. Added a testcase HiveHookIT.testDropNonExistingTable to test this. There are break statements missing after ALTERTABLE_LOCATION and DROPTABLE/DROPVIEW. While the latter are the last branches of the switch, it is still safer to add IMO. --> Added It appears the lineage query in the backend for a particular dataset (say inputs for this dataset), will not return just the process information if there is no corresponding dataset that is created as output of this process. So, maybe the truncate is not going to work as I have mentioned. Can you please verify? --> Verified in the HiveHookIT.testTruncateTable and observed the same that lineage doesnt work if inputs or outputs are not set for a process. We can fix it in two ways - Tracked at ATLAS-653 a. Fix the lineage for this because truncate table having any inputs does not make much sense to me. ( This is also the case with the HookContext being passed from hive - the context does not have any inputs) . The same behaviour is seen in insert to and from local directory, load from local directory queries( because we dont model the local filesystem paths ) b. Have truncate table having the same table as input and output . Model local file system paths for the other queries dealing with local paths was (Author: suma.shivaprasad): [~yhemanth] Thanks for the review. The handling of truncate table will create a single process with no inputs and a single table as output. We should validate if this type of lineage view is OK for a end user. I agree though that capturing the information is important. --> Raised ATLAS-653 to track this. On the same note, is the query string captured correctly for the process. Can we enhance the truncate test to validate this? --> HiveHookIT.validateProcess already check this by querying with the same exact query If a table / view being dropped is not present in Atlas, this would generate a 404 and we would loose the information. Maybe we can file a bug and track this for later? --> Raised ATLAS-652 to track this. We should also test what happens if a user executes a drop table if exists non_existent_table - do we get called? --> I observed that the hook is being called and the hook is ignoring this since there are no outputs for this query. I dont think we should capture these kind of queries . Please let me know if you feel otherwise. Added a testcase HiveHookIT.testDropNonExistingTable to test this. There are break statements missing after ALTERTABLE_LOCATION and DROPTABLE/DROPVIEW. While the latter are the last branches of the switch, it is still safer to add IMO. --> Added > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.1.patch, ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new sta
[jira] [Comment Edited] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230983#comment-15230983 ] Suma Shivaprasad edited comment on ATLAS-528 at 4/7/16 8:23 PM: [~yhemanth] Thanks for the review. The handling of truncate table will create a single process with no inputs and a single table as output. We should validate if this type of lineage view is OK for a end user. I agree though that capturing the information is important. --> Raised ATLAS-653 to track this. On the same note, is the query string captured correctly for the process. Can we enhance the truncate test to validate this? --> HiveHookIT.validateProcess already check this by querying with the same exact query If a table / view being dropped is not present in Atlas, this would generate a 404 and we would loose the information. Maybe we can file a bug and track this for later? --> Raised ATLAS-652 to track this. We should also test what happens if a user executes a drop table if exists non_existent_table - do we get called? --> I observed that the hook is being called and the hook is ignoring this since there are no outputs for this query. I dont think we should capture these kind of queries . Please let me know if you feel otherwise. Added a testcase HiveHookIT.testDropNonExistingTable to test this. There are break statements missing after ALTERTABLE_LOCATION and DROPTABLE/DROPVIEW. While the latter are the last branches of the switch, it is still safer to add IMO. --> Added was (Author: suma.shivaprasad): [~yhemanth] Thanks for the review. The handling of truncate table will create a single process with no inputs and a single table as output. We should validate if this type of lineage view is OK for a end user. I agree though that capturing the information is important. --> Raised ATLAS-653 to track this. On the same note, is the query string captured correctly for the process. Can we enhance the truncate test to validate this? --> HiveHookIT.validateProcess already check this by querying with the same exact query If a table / view being dropped is not present in Atlas, this would generate a 404 and we would loose the information. Maybe we can file a bug and track this for later? --> Raised ATLAS-652 to track this. We should also test what happens if a user executes a drop table if exists non_existent_table - do we get called? --> I observed that the hooks is being called and the hook is ignoring this since there are no outputs for this query. Added a testcase HiveHookIT.testDropNonExistingTable to test this. There are break statements missing after ALTERTABLE_LOCATION and DROPTABLE/DROPVIEW. While the latter are the last branches of the switch, it is still safer to add IMO. --> Added > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.1.patch, ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230983#comment-15230983 ] Suma Shivaprasad commented on ATLAS-528: @Hemanth yamijala Thanks for the review. The handling of truncate table will create a single process with no inputs and a single table as output. We should validate if this type of lineage view is OK for a end user. I agree though that capturing the information is important. --> Raised ATLAS-653 to track this. On the same note, is the query string captured correctly for the process. Can we enhance the truncate test to validate this? --> HiveHookIT.validateProcess already check this by querying with the same exact query If a table / view being dropped is not present in Atlas, this would generate a 404 and we would loose the information. Maybe we can file a bug and track this for later? --> Raised ATLAS-652 to track this. We should also test what happens if a user executes a drop table if exists non_existent_table - do we get called? --> I observed that the hooks is being called and the hook is ignoring this since there are no outputs for this query. Added a testcase HiveHookIT.testDropNonExistingTable to test this. There are break statements missing after ALTERTABLE_LOCATION and DROPTABLE/DROPVIEW. While the latter are the last branches of the switch, it is still safer to add IMO. --> Added > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.1.patch, ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230983#comment-15230983 ] Suma Shivaprasad edited comment on ATLAS-528 at 4/7/16 8:21 PM: [~yhemanth] Thanks for the review. The handling of truncate table will create a single process with no inputs and a single table as output. We should validate if this type of lineage view is OK for a end user. I agree though that capturing the information is important. --> Raised ATLAS-653 to track this. On the same note, is the query string captured correctly for the process. Can we enhance the truncate test to validate this? --> HiveHookIT.validateProcess already check this by querying with the same exact query If a table / view being dropped is not present in Atlas, this would generate a 404 and we would loose the information. Maybe we can file a bug and track this for later? --> Raised ATLAS-652 to track this. We should also test what happens if a user executes a drop table if exists non_existent_table - do we get called? --> I observed that the hooks is being called and the hook is ignoring this since there are no outputs for this query. Added a testcase HiveHookIT.testDropNonExistingTable to test this. There are break statements missing after ALTERTABLE_LOCATION and DROPTABLE/DROPVIEW. While the latter are the last branches of the switch, it is still safer to add IMO. --> Added was (Author: suma.shivaprasad): @Hemanth yamijala Thanks for the review. The handling of truncate table will create a single process with no inputs and a single table as output. We should validate if this type of lineage view is OK for a end user. I agree though that capturing the information is important. --> Raised ATLAS-653 to track this. On the same note, is the query string captured correctly for the process. Can we enhance the truncate test to validate this? --> HiveHookIT.validateProcess already check this by querying with the same exact query If a table / view being dropped is not present in Atlas, this would generate a 404 and we would loose the information. Maybe we can file a bug and track this for later? --> Raised ATLAS-652 to track this. We should also test what happens if a user executes a drop table if exists non_existent_table - do we get called? --> I observed that the hooks is being called and the hook is ignoring this since there are no outputs for this query. Added a testcase HiveHookIT.testDropNonExistingTable to test this. There are break statements missing after ALTERTABLE_LOCATION and DROPTABLE/DROPVIEW. While the latter are the last branches of the switch, it is still safer to add IMO. --> Added > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.1.patch, ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-528: --- Attachment: ATLAS-528.1.patch Attaching after fixing some of the review comments > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.1.patch, ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-653) Truncate table, insert into local dir, load from local dir does not show up in lineage
Suma Shivaprasad created ATLAS-653: -- Summary: Truncate table, insert into local dir, load from local dir does not show up in lineage Key: ATLAS-653 URL: https://issues.apache.org/jira/browse/ATLAS-653 Project: Atlas Issue Type: Bug Affects Versions: 0.7-incubating Reporter: Suma Shivaprasad Fix For: 0.7-incubating All the above commands do not show up in lineage since they dont have any input entities registered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-652) Delete on a non-existing table/view/db will result in 404
[ https://issues.apache.org/jira/browse/ATLAS-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230477#comment-15230477 ] Suma Shivaprasad commented on ATLAS-652: If drop on a table/db/db is done and it doesnt exist, we will lose the information about delete since it doesnt exist in atlas repository > Delete on a non-existing table/view/db will result in 404 > - > > Key: ATLAS-652 > URL: https://issues.apache.org/jira/browse/ATLAS-652 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-652) Delete on a non-existing table/view/db will result in 404
Suma Shivaprasad created ATLAS-652: -- Summary: Delete on a non-existing table/view/db will result in 404 Key: ATLAS-652 URL: https://issues.apache.org/jira/browse/ATLAS-652 Project: Atlas Issue Type: Bug Affects Versions: 0.7-incubating Reporter: Suma Shivaprasad Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-527) Support lineage for load table, import, export
[ https://issues.apache.org/jira/browse/ATLAS-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230416#comment-15230416 ] Suma Shivaprasad commented on ATLAS-527: Attached patch from RB which was +1ed > Support lineage for load table, import, export > -- > > Key: ATLAS-527 > URL: https://issues.apache.org/jira/browse/ATLAS-527 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-527.patch, ATLAS-527.patch > > > Currently lineage is not depicted from HDFS paths being loaded through these > commands to a hive table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-527) Support lineage for load table, import, export
[ https://issues.apache.org/jira/browse/ATLAS-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-527: --- Attachment: ATLAS-527.patch > Support lineage for load table, import, export > -- > > Key: ATLAS-527 > URL: https://issues.apache.org/jira/browse/ATLAS-527 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-527.patch, ATLAS-527.patch > > > Currently lineage is not depicted from HDFS paths being loaded through these > commands to a hive table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229324#comment-15229324 ] Suma Shivaprasad commented on ATLAS-528: Also have added support for truncate table, alter table partition column type as part of the same patch > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229315#comment-15229315 ] Suma Shivaprasad commented on ATLAS-528: This is dependent on patch of ATLAS-527 which is under review. Hence not making it Patch Available. > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-528: --- Attachment: ATLAS-528.patch > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad reassigned ATLAS-528: -- Assignee: Suma Shivaprasad > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-498) Support Embedded HBase
[ https://issues.apache.org/jira/browse/ATLAS-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15228556#comment-15228556 ] Suma Shivaprasad commented on ATLAS-498: +1 for the patch. Can you pls upload the latest patch after merging with trunk > Support Embedded HBase > -- > > Key: ATLAS-498 > URL: https://issues.apache.org/jira/browse/ATLAS-498 > Project: Atlas > Issue Type: New Feature >Affects Versions: 0.7-incubating >Reporter: Erik Bergenholtz >Assignee: Tom Beerbower > Fix For: 0.7-incubating > > Attachments: ATLAS-498.patch > > > If possible, Atlas should switch over to use embedded HBase. Initially, this > is an investigative task which it if proves successful with result in > switching default Atlas configuration/deployment to use embedded HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-642) import hive should also create the lineage for external tables
Suma Shivaprasad created ATLAS-642: -- Summary: import hive should also create the lineage for external tables Key: ATLAS-642 URL: https://issues.apache.org/jira/browse/ATLAS-642 Project: Atlas Issue Type: Sub-task Reporter: Suma Shivaprasad import-hive can also start showing lineage between HDFS paths and tables. However the create table process should also register the exact query that was used to create the table. Show create table does that by reconstructing the query. Need to figure out if we can potentially use that to create the query -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-527) Support lineage for load table, import, export
[ https://issues.apache.org/jira/browse/ATLAS-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-527: --- Attachment: ATLAS-527.patch > Support lineage for load table, import, export > -- > > Key: ATLAS-527 > URL: https://issues.apache.org/jira/browse/ATLAS-527 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-527.patch > > > Currently lineage is not depicted from HDFS paths being loaded through these > commands to a hive table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-527) Support lineage for load table, import, export
[ https://issues.apache.org/jira/browse/ATLAS-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15227389#comment-15227389 ] Suma Shivaprasad commented on ATLAS-527: https://reviews.apache.org/r/45784/ > Support lineage for load table, import, export > -- > > Key: ATLAS-527 > URL: https://issues.apache.org/jira/browse/ATLAS-527 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > > Currently lineage is not depicted from HDFS paths being loaded through these > commands to a hive table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-572) Handle secure instance of Zookeeper for leader election.
[ https://issues.apache.org/jira/browse/ATLAS-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15227374#comment-15227374 ] Suma Shivaprasad commented on ATLAS-572: +1 > Handle secure instance of Zookeeper for leader election. > > > Key: ATLAS-572 > URL: https://issues.apache.org/jira/browse/ATLAS-572 > Project: Atlas > Issue Type: Sub-task >Reporter: Hemanth Yamijala >Assignee: Hemanth Yamijala > Attachments: ATLAS-572-1.patch, ATLAS-572-2.patch, ATLAS-572.patch > > > ATLAS-511 introduces Zookeeper based leader election in unsecure mode. In > this JIRA we will make necessary changes for making Atlas work with secure > Zookeeper installations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-638) UI : Recognize and display HDFS Paths with a different notation
Suma Shivaprasad created ATLAS-638: -- Summary: UI : Recognize and display HDFS Paths with a different notation Key: ATLAS-638 URL: https://issues.apache.org/jira/browse/ATLAS-638 Project: Atlas Issue Type: New Feature Reporter: Suma Shivaprasad Need to differentiate between tables and HDFS paths -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-572) Handle secure instance of Zookeeper for leader election.
[ https://issues.apache.org/jira/browse/ATLAS-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225517#comment-15225517 ] Suma Shivaprasad commented on ATLAS-572: Also another question I had was, typically components like hbase have a separate zkRoot for secure and unsecure mode. Is it the same case with the curator created ZK paths? > Handle secure instance of Zookeeper for leader election. > > > Key: ATLAS-572 > URL: https://issues.apache.org/jira/browse/ATLAS-572 > Project: Atlas > Issue Type: Sub-task >Reporter: Hemanth Yamijala >Assignee: Hemanth Yamijala > Attachments: ATLAS-572.patch > > > ATLAS-511 introduces Zookeeper based leader election in unsecure mode. In > this JIRA we will make necessary changes for making Atlas work with secure > Zookeeper installations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-572) Handle secure instance of Zookeeper for leader election.
[ https://issues.apache.org/jira/browse/ATLAS-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225515#comment-15225515 ] Suma Shivaprasad commented on ATLAS-572: Couple of minor review comments 1. Instead of us validating the auth/acls, is there any way of doing it through ZK code? Its fine, even if we dont do this now. Not a must do comment. 2. Log the acl, auth and current user principal > Handle secure instance of Zookeeper for leader election. > > > Key: ATLAS-572 > URL: https://issues.apache.org/jira/browse/ATLAS-572 > Project: Atlas > Issue Type: Sub-task >Reporter: Hemanth Yamijala >Assignee: Hemanth Yamijala > Attachments: ATLAS-572.patch > > > ATLAS-511 introduces Zookeeper based leader election in unsecure mode. In > this JIRA we will make necessary changes for making Atlas work with secure > Zookeeper installations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-605) Hook Notifications for DELETE entity needs to be supported
[ https://issues.apache.org/jira/browse/ATLAS-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-605: --- Attachment: ATLAS-605.2.patch Fixed merge issue and ran build which succeeded > Hook Notifications for DELETE entity needs to be supported > -- > > Key: ATLAS-605 > URL: https://issues.apache.org/jira/browse/ATLAS-605 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-605.1.patch, ATLAS-605.2.patch, ATLAS-605.patch > > > HookNotificationType needs to have a separate DELETE operation and hook > notification messages should be able to use this operation for any deletes > Can be applied only after ATLAS-607 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-605) Hook Notifications for DELETE entity needs to be supported
[ https://issues.apache.org/jira/browse/ATLAS-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-605: --- Attachment: ATLAS-605.1.patch Merged with trunk and Fixed review comments > Hook Notifications for DELETE entity needs to be supported > -- > > Key: ATLAS-605 > URL: https://issues.apache.org/jira/browse/ATLAS-605 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: ATLAS-605.1.patch, ATLAS-605.patch > > > HookNotificationType needs to have a separate DELETE operation and hook > notification messages should be able to use this operation for any deletes > Can be applied only after ATLAS-607 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-607) Add Support for delete entity through a qualifiedName
[ https://issues.apache.org/jira/browse/ATLAS-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-607: --- Attachment: ATLAS-607.4.patch Merged with trunk and changed to a better a log message for EntityNotFoundException > Add Support for delete entity through a qualifiedName > - > > Key: ATLAS-607 > URL: https://issues.apache.org/jira/browse/ATLAS-607 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-607.1.patch, ATLAS-607.2.patch, ATLAS-607.3.patch, > ATLAS-607.4.patch, ATLAS-607.patch > > > Currently deleteEntity supports deletion only by guids. Need to support > deletion by an entities' unique attribute as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-607) Add Support for delete entity through a qualifiedName
[ https://issues.apache.org/jira/browse/ATLAS-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-607: --- Attachment: ATLAS-607.3.patch Merged with trunk > Add Support for delete entity through a qualifiedName > - > > Key: ATLAS-607 > URL: https://issues.apache.org/jira/browse/ATLAS-607 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-607.1.patch, ATLAS-607.2.patch, ATLAS-607.3.patch, > ATLAS-607.patch > > > Currently deleteEntity supports deletion only by guids. Need to support > deletion by an entities' unique attribute as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-607) Add Support for delete entity through a qualifiedName
[ https://issues.apache.org/jira/browse/ATLAS-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-607: --- Attachment: ATLAS-607.2.patch Fixed review comments > Add Support for delete entity through a qualifiedName > - > > Key: ATLAS-607 > URL: https://issues.apache.org/jira/browse/ATLAS-607 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-607.1.patch, ATLAS-607.2.patch, ATLAS-607.patch > > > Currently deleteEntity supports deletion only by guids. Need to support > deletion by an entities' unique attribute as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (ATLAS-627) Change column name should retain traits/tags assigned to columns
[ https://issues.apache.org/jira/browse/ATLAS-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad resolved ATLAS-627. Resolution: Duplicate of ATLAS-583 > Change column name should retain traits/tags assigned to columns > > > Key: ATLAS-627 > URL: https://issues.apache.org/jira/browse/ATLAS-627 > Project: Atlas > Issue Type: Bug >Affects Versions: trunk >Reporter: Ayub Khan > Fix For: trunk > > > Currently changing the column name results in losing the trait attached to > it. Ideally trait should be retained even after renaming the column. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-605) Hook Notifications for DELETE entity needs to be supported
[ https://issues.apache.org/jira/browse/ATLAS-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-605: --- Description: HookNotificationType needs to have a separate DELETE operation and hook notification messages should be able to use this operation for any deletes Can be applied only after ATLAS-607 was:HookNotificationType needs to have a separate DELETE operation and hook notification messages should be able to use this operation for any deletes > Hook Notifications for DELETE entity needs to be supported > -- > > Key: ATLAS-605 > URL: https://issues.apache.org/jira/browse/ATLAS-605 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: ATLAS-605.patch > > > HookNotificationType needs to have a separate DELETE operation and hook > notification messages should be able to use this operation for any deletes > Can be applied only after ATLAS-607 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-605) Hook Notifications for DELETE entity needs to be supported
[ https://issues.apache.org/jira/browse/ATLAS-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-605: --- Attachment: ATLAS-605.patch > Hook Notifications for DELETE entity needs to be supported > -- > > Key: ATLAS-605 > URL: https://issues.apache.org/jira/browse/ATLAS-605 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: ATLAS-605.patch > > > HookNotificationType needs to have a separate DELETE operation and hook > notification messages should be able to use this operation for any deletes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-607) Add Support for delete entity through a qualifiedName
[ https://issues.apache.org/jira/browse/ATLAS-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-607: --- Attachment: ATLAS-607.1.patch > Add Support for delete entity through a qualifiedName > - > > Key: ATLAS-607 > URL: https://issues.apache.org/jira/browse/ATLAS-607 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-607.1.patch, ATLAS-607.patch > > > Currently deleteEntity supports deletion only by guids. Need to support > deletion by an entities' unique attribute as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-607) Add Support for delete entity through a qualifiedName
[ https://issues.apache.org/jira/browse/ATLAS-607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222745#comment-15222745 ] Suma Shivaprasad commented on ATLAS-607: https://reviews.apache.org/r/45613 > Add Support for delete entity through a qualifiedName > - > > Key: ATLAS-607 > URL: https://issues.apache.org/jira/browse/ATLAS-607 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-607.patch > > > Currently deleteEntity supports deletion only by guids. Need to support > deletion by an entities' unique attribute as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-619) Partition queries should be normalized when tracking lineage at table
Suma Shivaprasad created ATLAS-619: -- Summary: Partition queries should be normalized when tracking lineage at table Key: ATLAS-619 URL: https://issues.apache.org/jira/browse/ATLAS-619 Project: Atlas Issue Type: Bug Affects Versions: 0.7-incubating Reporter: Suma Shivaprasad Partition type and entities are no loner being created after ATLAS-599. Even prior to this change , the linage for partition based queries are being tracked at the table level and not at the partition level which would be ideal. To track at the table level, the queries should be canonicialized for the partition values or only the latest few partition queries should be tracked since otherwise there could be potentially thousands of such queries while tracking lineage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (ATLAS-605) Hook Notifications for DELETE entity needs to be supported
[ https://issues.apache.org/jira/browse/ATLAS-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad reassigned ATLAS-605: -- Assignee: Suma Shivaprasad > Hook Notifications for DELETE entity needs to be supported > -- > > Key: ATLAS-605 > URL: https://issues.apache.org/jira/browse/ATLAS-605 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > > HookNotificationType needs to have a separate DELETE operation and hook > notification messages should be able to use this operation for any deletes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-607) Add Support for delete entity through a qualifiedName
[ https://issues.apache.org/jira/browse/ATLAS-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-607: --- Attachment: ATLAS-607.patch > Add Support for delete entity through a qualifiedName > - > > Key: ATLAS-607 > URL: https://issues.apache.org/jira/browse/ATLAS-607 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: ATLAS-607.patch > > > Currently deleteEntity supports deletion only by guids. Need to support > deletion by an entities' unique attribute as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (ATLAS-607) Add Support for delete entity through a qualifiedName
[ https://issues.apache.org/jira/browse/ATLAS-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad reassigned ATLAS-607: -- Assignee: Suma Shivaprasad > Add Support for delete entity through a qualifiedName > - > > Key: ATLAS-607 > URL: https://issues.apache.org/jira/browse/ATLAS-607 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: ATLAS-607.patch > > > Currently deleteEntity supports deletion only by guids. Need to support > deletion by an entities' unique attribute as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-618) Fix assembly for hdfs-module
[ https://issues.apache.org/jira/browse/ATLAS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-618: --- Attachment: ATLAS-618.patch > Fix assembly for hdfs-module > > > Key: ATLAS-618 > URL: https://issues.apache.org/jira/browse/ATLAS-618 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-618.patch > > > standalone-package.xml does not have the assembly configuration for > hdfs-module. > Also all the hooks need this dependency added in the pom -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-618) Fix assembly for hdfs-module
[ https://issues.apache.org/jira/browse/ATLAS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-618: --- Description: standalone-package.xml does not have the assembly configuration for hdfs-module. Also all the hooks need this dependency added in the pom was:standalone-package.xml does not have the assembly configuration for hdfs-module. Needs to be added > Fix assembly for hdfs-module > > > Key: ATLAS-618 > URL: https://issues.apache.org/jira/browse/ATLAS-618 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > > standalone-package.xml does not have the assembly configuration for > hdfs-module. > Also all the hooks need this dependency added in the pom -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-618) Fix assembly for hdfs-module
[ https://issues.apache.org/jira/browse/ATLAS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-618: --- Description: standalone-package.xml does not have the assembly configuration for hdfs-module. Needs to be added > Fix assembly for hdfs-module > > > Key: ATLAS-618 > URL: https://issues.apache.org/jira/browse/ATLAS-618 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > > standalone-package.xml does not have the assembly configuration for > hdfs-module. Needs to be added -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-618) Fix assembly for hdfs-module
Suma Shivaprasad created ATLAS-618: -- Summary: Fix assembly for hdfs-module Key: ATLAS-618 URL: https://issues.apache.org/jira/browse/ATLAS-618 Project: Atlas Issue Type: Bug Affects Versions: 0.7-incubating Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Fix For: 0.7-incubating -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-525) Drop support for partitions, select query lineage, roles, principals, resource, hive_type...
[ https://issues.apache.org/jira/browse/ATLAS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-525: --- Attachment: ATLAS-525.5.patch Rebased after ATLAS-599 commit > Drop support for partitions, select query lineage, roles, principals, > resource, hive_type... > > > Key: ATLAS-525 > URL: https://issues.apache.org/jira/browse/ATLAS-525 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-525.2.patch, ATLAS-525.3.patch, ATLAS-525.4.patch, > ATLAS-525.5.patch, ATLAS-525.patch > > > Currently partition entities are being created upon partition addition and > whenever a select, a hive_process is created which we are not making any use > of later for lineage. The proposal is to drop support for partitions and drop > tracking lineage for select queries until we figure out how to make > meaningful use of this data. > Remove hive_role ,hive_type, hive_resource, hive_object from model since it > is not being used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-599) HDFS Path Model
[ https://issues.apache.org/jira/browse/ATLAS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-599: --- Attachment: ATLAS-599.3.patch Attaching the latest patch after rebasing with trunk > HDFS Path Model > > > Key: ATLAS-599 > URL: https://issues.apache.org/jira/browse/ATLAS-599 > Project: Atlas > Issue Type: New Feature >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-599.1.patch, ATLAS-599.2.patch, ATLAS-599.3.patch, > ATLAS-599.patch > > > Model HDFS paths as a DataSet so that lineage could be tracked for them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-553) Entity mutation - Fix issue with reordering of elements in array with composite references
[ https://issues.apache.org/jira/browse/ATLAS-553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222139#comment-15222139 ] Suma Shivaprasad commented on ATLAS-553: Committed. Thanks for the review [~shwethags] > Entity mutation - Fix issue with reordering of elements in array with > composite references > - > > Key: ATLAS-553 > URL: https://issues.apache.org/jira/browse/ATLAS-553 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-553.1.patch, ATLAS-553.2.patch, ATLAS-553.patch > > > Currently, if theres an update on the entity to the order of elements in an > array/aray etc, it is not handled correctly since entity > mutation logic thinks that the element is deleted resulting in a deleted > vertex being referred later causing entity mutation to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-513) Admin support for HA
[ https://issues.apache.org/jira/browse/ATLAS-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222088#comment-15222088 ] Suma Shivaprasad commented on ATLAS-513: Committed. Thanks [~yhemanth] > Admin support for HA > > > Key: ATLAS-513 > URL: https://issues.apache.org/jira/browse/ATLAS-513 > Project: Atlas > Issue Type: Sub-task >Reporter: Hemanth Yamijala >Assignee: Hemanth Yamijala > Fix For: 0.7-incubating > > Attachments: ATLAS-513-1.patch, ATLAS-513.patch > > > The following enhancements are proposed for the admin interface to manage HA: > * Status of any atlas server instance via API and Atlas Admin UI. > * Admin API to switch from active to passive. > * Admin switch from passive to active - this must fail if there is another > active. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-525) Drop support for partitions, select query lineage, roles, principals, resource, hive_type...
[ https://issues.apache.org/jira/browse/ATLAS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-525: --- Attachment: ATLAS-525.4.patch Attaching patch after addressing all review comments > Drop support for partitions, select query lineage, roles, principals, > resource, hive_type... > > > Key: ATLAS-525 > URL: https://issues.apache.org/jira/browse/ATLAS-525 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-525.2.patch, ATLAS-525.3.patch, ATLAS-525.4.patch, > ATLAS-525.patch > > > Currently partition entities are being created upon partition addition and > whenever a select, a hive_process is created which we are not making any use > of later for lineage. The proposal is to drop support for partitions and drop > tracking lineage for select queries until we figure out how to make > meaningful use of this data. > Remove hive_role ,hive_type, hive_resource, hive_object from model since it > is not being used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-496) Optimistic locking to prevent update conflicts
[ https://issues.apache.org/jira/browse/ATLAS-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221960#comment-15221960 ] Suma Shivaprasad commented on ATLAS-496: [~dkantor] Are you proposing to have a distributed lock across multiple servers in this jira? Currently entity updates within a single server are already protected with a inter-thread lock though it is limited to Hbase.(not a 2PC across Hbase and Solr) Type updates need to be looked into though like [~jonesn] mentioned > Optimistic locking to prevent update conflicts > -- > > Key: ATLAS-496 > URL: https://issues.apache.org/jira/browse/ATLAS-496 > Project: Atlas > Issue Type: Improvement >Reporter: David Kantor > > The ATLAS api does not adequately protect against concurrent activity against > the same entity, type or other object. > Some form of locking is desirable, though strict locking and/or 2pc are > problematic in a highly distributed environment. Optimistic locking may be > the best approach so that at a minimum an application attempting to do a > modification will be alerted if another application has touched the same > (object) since that app last retrieved the object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-525) Drop support for partitions, select query lineage, roles, principals, resource, hive_type...
[ https://issues.apache.org/jira/browse/ATLAS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-525: --- Attachment: ATLAS-525.3.patch > Drop support for partitions, select query lineage, roles, principals, > resource, hive_type... > > > Key: ATLAS-525 > URL: https://issues.apache.org/jira/browse/ATLAS-525 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-525.2.patch, ATLAS-525.3.patch, ATLAS-525.patch > > > Currently partition entities are being created upon partition addition and > whenever a select, a hive_process is created which we are not making any use > of later for lineage. The proposal is to drop support for partitions and drop > tracking lineage for select queries until we figure out how to make > meaningful use of this data. > Remove hive_role ,hive_type, hive_resource, hive_object from model since it > is not being used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-525) Drop support for partitions, select query lineage, roles, principals, resource, hive_type...
[ https://issues.apache.org/jira/browse/ATLAS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-525: --- Attachment: ATLAS-525.2.patch > Drop support for partitions, select query lineage, roles, principals, > resource, hive_type... > > > Key: ATLAS-525 > URL: https://issues.apache.org/jira/browse/ATLAS-525 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-525.2.patch, ATLAS-525.patch > > > Currently partition entities are being created upon partition addition and > whenever a select, a hive_process is created which we are not making any use > of later for lineage. The proposal is to drop support for partitions and drop > tracking lineage for select queries until we figure out how to make > meaningful use of this data. > Remove hive_role ,hive_type, hive_resource, hive_object from model since it > is not being used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-553) Entity mutation - Fix issue with reordering of elements in array with composite references
[ https://issues.apache.org/jira/browse/ATLAS-553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-553: --- Attachment: ATLAS-553.2.patch Attaching rebased patch > Entity mutation - Fix issue with reordering of elements in array with > composite references > - > > Key: ATLAS-553 > URL: https://issues.apache.org/jira/browse/ATLAS-553 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-553.1.patch, ATLAS-553.2.patch, ATLAS-553.patch > > > Currently, if theres an update on the entity to the order of elements in an > array/aray etc, it is not handled correctly since entity > mutation logic thinks that the element is deleted resulting in a deleted > vertex being referred later causing entity mutation to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-573) Inherited attributes disappear from entities after server restart
[ https://issues.apache.org/jira/browse/ATLAS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220982#comment-15220982 ] Suma Shivaprasad commented on ATLAS-573: +1 > Inherited attributes disappear from entities after server restart > - > > Key: ATLAS-573 > URL: https://issues.apache.org/jira/browse/ATLAS-573 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.6-incubating, 0.7-incubating >Reporter: David Kantor >Assignee: David Kantor >Priority: Blocker > Attachments: rb45195.patch > > > After server restart, attributes that are inherited from a superclass are not > included in the representation of an entity when retrieved from the > repository. It appears to be an issue with how the type system is loaded > from the repository at server startup (GraphBackedTypeStore.restore(), such > that the field mappings for subclasses do not include the attributes from > superclasses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-513) Admin support for HA
[ https://issues.apache.org/jira/browse/ATLAS-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219409#comment-15219409 ] Suma Shivaprasad commented on ATLAS-513: +1. LGTM > Admin support for HA > > > Key: ATLAS-513 > URL: https://issues.apache.org/jira/browse/ATLAS-513 > Project: Atlas > Issue Type: Sub-task >Reporter: Hemanth Yamijala >Assignee: Hemanth Yamijala > Attachments: ATLAS-513.patch > > > The following enhancements are proposed for the admin interface to manage HA: > * Status of any atlas server instance via API and Atlas Admin UI. > * Admin API to switch from active to passive. > * Admin switch from passive to active - this must fail if there is another > active. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-599) HDFS Path Model
[ https://issues.apache.org/jira/browse/ATLAS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219373#comment-15219373 ] Suma Shivaprasad commented on ATLAS-599: https://reviews.apache.org/r/45452/ > HDFS Path Model > > > Key: ATLAS-599 > URL: https://issues.apache.org/jira/browse/ATLAS-599 > Project: Atlas > Issue Type: New Feature >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-599.1.patch, ATLAS-599.2.patch, ATLAS-599.patch > > > Model HDFS paths as a DataSet so that lineage could be tracked for them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-525) Drop support for partitions, select query lineage, roles, principals, resource, hive_type...
[ https://issues.apache.org/jira/browse/ATLAS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219371#comment-15219371 ] Suma Shivaprasad commented on ATLAS-525: https://reviews.apache.org/r/45526/ > Drop support for partitions, select query lineage, roles, principals, > resource, hive_type... > > > Key: ATLAS-525 > URL: https://issues.apache.org/jira/browse/ATLAS-525 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-525.patch > > > Currently partition entities are being created upon partition addition and > whenever a select, a hive_process is created which we are not making any use > of later for lineage. The proposal is to drop support for partitions and drop > tracking lineage for select queries until we figure out how to make > meaningful use of this data. > Remove hive_role ,hive_type, hive_resource, hive_object from model since it > is not being used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-525) Drop support for partitions, select query lineage, roles, principals, resource, hive_type...
[ https://issues.apache.org/jira/browse/ATLAS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219370#comment-15219370 ] Suma Shivaprasad commented on ATLAS-525: This patch covers the following cases 1. Remove partition from model, entity creation and also stop creating a hive_process for any partition based queries like insert into/overwrite partition, load into partition etc 2. Also filters out select queries by not creating the hive_process for them 3. Removed the other types from model - hive_role ,hive_type, hive_resource, hive_object which were not used 4. Added tests for ignoring selects, ignoring partition based queries for load, insert > Drop support for partitions, select query lineage, roles, principals, > resource, hive_type... > > > Key: ATLAS-525 > URL: https://issues.apache.org/jira/browse/ATLAS-525 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-525.patch > > > Currently partition entities are being created upon partition addition and > whenever a select, a hive_process is created which we are not making any use > of later for lineage. The proposal is to drop support for partitions and drop > tracking lineage for select queries until we figure out how to make > meaningful use of this data. > Remove hive_role ,hive_type, hive_resource, hive_object from model since it > is not being used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-525) Drop support for partitions, select query lineage, roles, principals, resource, hive_type...
[ https://issues.apache.org/jira/browse/ATLAS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-525: --- Description: Currently partition entities are being created upon partition addition and whenever a select, a hive_process is created which we are not making any use of later for lineage. The proposal is to drop support for partitions and drop tracking lineage for select queries until we figure out how to make meaningful use of this data. Remove hive_role ,hive_type, hive_resource, hive_object from model since it is not being used was: Currently partitions and select queries are being created upon partition addition and whenever a select is run which we are not making use of later for lineage. The proposal is to drop support for partitions and drop tracking lineage for select queries until we figure out how to make meaningful use of this data. Remove hive_role , hive_principal ,hive_type, hive_resource, hive_object, hive_index, from model since it is not being used > Drop support for partitions, select query lineage, roles, principals, > resource, hive_type... > > > Key: ATLAS-525 > URL: https://issues.apache.org/jira/browse/ATLAS-525 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-525.patch > > > Currently partition entities are being created upon partition addition and > whenever a select, a hive_process is created which we are not making any use > of later for lineage. The proposal is to drop support for partitions and drop > tracking lineage for select queries until we figure out how to make > meaningful use of this data. > Remove hive_role ,hive_type, hive_resource, hive_object from model since it > is not being used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-525) Drop support for partitions, select query lineage, roles, principals, resource, hive_type...
[ https://issues.apache.org/jira/browse/ATLAS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-525: --- Attachment: ATLAS-525.patch > Drop support for partitions, select query lineage, roles, principals, > resource, hive_type... > > > Key: ATLAS-525 > URL: https://issues.apache.org/jira/browse/ATLAS-525 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-525.patch > > > Currently partitions and select queries are being created upon partition > addition and whenever a select is run which we are not making use of later > for lineage. The proposal is to drop support for partitions and drop tracking > lineage for select queries until we figure out how to make meaningful use of > this data. > Remove hive_role , hive_principal ,hive_type, hive_resource, hive_object, > hive_index, > from model since it is not being used -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-607) Add Support for delete entity through a qualifiedName
[ https://issues.apache.org/jira/browse/ATLAS-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-607: --- Description: Currently deleteEntity supports deletion only by guids. Need to support deletion by an entities' unique attribute as well. (was: Currently deleteEntity supports deletion only by guids. Need to support deletion by an entities' qualifiedName as well.) > Add Support for delete entity through a qualifiedName > - > > Key: ATLAS-607 > URL: https://issues.apache.org/jira/browse/ATLAS-607 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad > > Currently deleteEntity supports deletion only by guids. Need to support > deletion by an entities' unique attribute as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-607) Add Support for delete entity through a qualifiedName
[ https://issues.apache.org/jira/browse/ATLAS-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-607: --- Description: Currently deleteEntity supports deletion only by guids. Need to support deletion by an entities' qualifiedName as well. > Add Support for delete entity through a qualifiedName > - > > Key: ATLAS-607 > URL: https://issues.apache.org/jira/browse/ATLAS-607 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad > > Currently deleteEntity supports deletion only by guids. Need to support > deletion by an entities' qualifiedName as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-607) Add Support for delete entity through a qualifiedName
Suma Shivaprasad created ATLAS-607: -- Summary: Add Support for delete entity through a qualifiedName Key: ATLAS-607 URL: https://issues.apache.org/jira/browse/ATLAS-607 Project: Atlas Issue Type: Sub-task Reporter: Suma Shivaprasad -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-606) Change from Hive thrift interface to Hive MetastoreClient
Suma Shivaprasad created ATLAS-606: -- Summary: Change from Hive thrift interface to Hive MetastoreClient Key: ATLAS-606 URL: https://issues.apache.org/jira/browse/ATLAS-606 Project: Atlas Issue Type: Sub-task Reporter: Suma Shivaprasad The reason to switch would be a. Failovers from metastore client are not handled in Hive client b. Hive thrift calls are apparently unsecure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-605) Hook Notifications for DELETE entity needs to be supported
Suma Shivaprasad created ATLAS-605: -- Summary: Hook Notifications for DELETE entity needs to be supported Key: ATLAS-605 URL: https://issues.apache.org/jira/browse/ATLAS-605 Project: Atlas Issue Type: Sub-task Reporter: Suma Shivaprasad HookNotificationType needs to have a separate DELETE operation and hook notification messages should be able to use this operation for any deletes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (ATLAS-122) Support for Deletion of Entities
[ https://issues.apache.org/jira/browse/ATLAS-122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad reopened ATLAS-122: Reopening the issue since hook notifications for deletes also needs to be implemented. Raising a separate task under this bucket for tracking that. > Support for Deletion of Entities > > > Key: ATLAS-122 > URL: https://issues.apache.org/jira/browse/ATLAS-122 > Project: Atlas > Issue Type: New Feature >Reporter: Suma Shivaprasad >Assignee: David Kantor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-599) HDFS Path Model
[ https://issues.apache.org/jira/browse/ATLAS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-599: --- Attachment: ATLAS-599.2.patch Added missing entity attributes in Storm Hook > HDFS Path Model > > > Key: ATLAS-599 > URL: https://issues.apache.org/jira/browse/ATLAS-599 > Project: Atlas > Issue Type: New Feature >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-599.1.patch, ATLAS-599.2.patch, ATLAS-599.patch > > > Model HDFS paths as a DataSet so that lineage could be tracked for them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-599) HDFS Path Model
[ https://issues.apache.org/jira/browse/ATLAS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-599: --- Attachment: ATLAS-599.1.patch > HDFS Path Model > > > Key: ATLAS-599 > URL: https://issues.apache.org/jira/browse/ATLAS-599 > Project: Atlas > Issue Type: New Feature >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-599.1.patch, ATLAS-599.patch > > > Model HDFS paths as a DataSet so that lineage could be tracked for them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-599) HDFS Path Model
[ https://issues.apache.org/jira/browse/ATLAS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-599: --- Attachment: (was: ATLAS-599.1.patch) > HDFS Path Model > > > Key: ATLAS-599 > URL: https://issues.apache.org/jira/browse/ATLAS-599 > Project: Atlas > Issue Type: New Feature >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-599.patch > > > Model HDFS paths as a DataSet so that lineage could be tracked for them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-599) HDFS Path Model
[ https://issues.apache.org/jira/browse/ATLAS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216914#comment-15216914 ] Suma Shivaprasad commented on ATLAS-599: Fixed RAT check and build failure > HDFS Path Model > > > Key: ATLAS-599 > URL: https://issues.apache.org/jira/browse/ATLAS-599 > Project: Atlas > Issue Type: New Feature >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-599.1.patch, ATLAS-599.patch > > > Model HDFS paths as a DataSet so that lineage could be tracked for them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-599) HDFS Path Model
[ https://issues.apache.org/jira/browse/ATLAS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-599: --- Attachment: ATLAS-599.1.patch > HDFS Path Model > > > Key: ATLAS-599 > URL: https://issues.apache.org/jira/browse/ATLAS-599 > Project: Atlas > Issue Type: New Feature >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-599.1.patch, ATLAS-599.patch > > > Model HDFS paths as a DataSet so that lineage could be tracked for them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-599) HDFS Path Model
[ https://issues.apache.org/jira/browse/ATLAS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-599: --- Attachment: ATLAS-599.patch > HDFS Path Model > > > Key: ATLAS-599 > URL: https://issues.apache.org/jira/browse/ATLAS-599 > Project: Atlas > Issue Type: New Feature >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-599.patch > > > Model HDFS paths as a DataSet so that lineage could be tracked for them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-599) HDFS Path Model
Suma Shivaprasad created ATLAS-599: -- Summary: HDFS Path Model Key: ATLAS-599 URL: https://issues.apache.org/jira/browse/ATLAS-599 Project: Atlas Issue Type: New Feature Affects Versions: 0.7-incubating Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Fix For: 0.7-incubating Model HDFS paths as a DataSet so that lineage could be tracked for them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-595) 'Description/Comment' attribute needs to be normalized hive_db, hive_table
[ https://issues.apache.org/jira/browse/ATLAS-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-595: --- Summary: 'Description/Comment' attribute needs to be normalized hive_db, hive_table (was: 'Comment' attribute is not captured in hive_db) > 'Description/Comment' attribute needs to be normalized hive_db, hive_table > -- > > Key: ATLAS-595 > URL: https://issues.apache.org/jira/browse/ATLAS-595 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > > Database comment is not captured in the model -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-597) Cluster Name should be changed to cluster Id
Suma Shivaprasad created ATLAS-597: -- Summary: Cluster Name should be changed to cluster Id Key: ATLAS-597 URL: https://issues.apache.org/jira/browse/ATLAS-597 Project: Atlas Issue Type: Sub-task Reporter: Suma Shivaprasad Ambari Cluster Name or User specified cluster name could easily be changed by users and cluster should be represented by an id which is constant and tied to clusterName which could change -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ATLAS-595) 'Comment' attribute is not captured in hive_db
[ https://issues.apache.org/jira/browse/ATLAS-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210738#comment-15210738 ] Suma Shivaprasad edited comment on ATLAS-595 at 3/24/16 9:56 PM: - This is being captured but instead in the 'description' attribute . Could be renamed to 'comment' instead Also hive_table has a description attribute which is never used. These should be normalized and table 'description' could have the comment or comment could be normalized across database and table. was (Author: suma.shivaprasad): This is being captured but instead in the 'description' attribute . Could be renamed to 'comment' instead > 'Comment' attribute is not captured in hive_db > -- > > Key: ATLAS-595 > URL: https://issues.apache.org/jira/browse/ATLAS-595 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > > Database comment is not captured in the model -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-596) UI : Bucket cols and sortCols within storage descriptor are not displayed correctly
[ https://issues.apache.org/jira/browse/ATLAS-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-596: --- Description: Backend is returning data correctly. UI is not displaying the bucketCols and sortCols within storageDescript correctly.NUmBuckets is being shown as date instead of long. Attaching screenshot (was: Backend is returning data correctly. UI is not displaying the bucketCols and sortCols within storageDescript correctly. Attaching screenshot) > UI : Bucket cols and sortCols within storage descriptor are not displayed > correctly > --- > > Key: ATLAS-596 > URL: https://issues.apache.org/jira/browse/ATLAS-596 > Project: Atlas > Issue Type: Bug >Reporter: Suma Shivaprasad > Attachments: Screen Shot 2016-03-24 at 2.47.04 PM.png > > > Backend is returning data correctly. UI is not displaying the bucketCols and > sortCols within storageDescript correctly.NUmBuckets is being shown as date > instead of long. Attaching screenshot -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-596) UI : Bucket cols and sortCols within storage descriptor are not displayed correctly
Suma Shivaprasad created ATLAS-596: -- Summary: UI : Bucket cols and sortCols within storage descriptor are not displayed correctly Key: ATLAS-596 URL: https://issues.apache.org/jira/browse/ATLAS-596 Project: Atlas Issue Type: Bug Reporter: Suma Shivaprasad Attachments: Screen Shot 2016-03-24 at 2.47.04 PM.png Backend is returning data correctly. UI is not displaying the bucketCols and sortCols within storageDescript correctly. Attaching screenshot -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-596) UI : Bucket cols and sortCols within storage descriptor are not displayed correctly
[ https://issues.apache.org/jira/browse/ATLAS-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-596: --- Attachment: Screen Shot 2016-03-24 at 2.47.04 PM.png > UI : Bucket cols and sortCols within storage descriptor are not displayed > correctly > --- > > Key: ATLAS-596 > URL: https://issues.apache.org/jira/browse/ATLAS-596 > Project: Atlas > Issue Type: Bug >Reporter: Suma Shivaprasad > Attachments: Screen Shot 2016-03-24 at 2.47.04 PM.png > > > Backend is returning data correctly. UI is not displaying the bucketCols and > sortCols within storageDescript correctly. Attaching screenshot -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-595) 'Comment' attribute is not captured in hive_db
[ https://issues.apache.org/jira/browse/ATLAS-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210738#comment-15210738 ] Suma Shivaprasad commented on ATLAS-595: This is being captured but instead in the 'description' attribute . Could be renamed to 'comment' instead > 'Comment' attribute is not captured in hive_db > -- > > Key: ATLAS-595 > URL: https://issues.apache.org/jira/browse/ATLAS-595 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > > Database comment is not captured in the model -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-595) 'Comment' attribute is not captured in hive_db
Suma Shivaprasad created ATLAS-595: -- Summary: 'Comment' attribute is not captured in hive_db Key: ATLAS-595 URL: https://issues.apache.org/jira/browse/ATLAS-595 Project: Atlas Issue Type: Sub-task Affects Versions: 0.7-incubating Reporter: Suma Shivaprasad Database comment is not captured in the model -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-594) alter table rename doesnt work across databases
Suma Shivaprasad created ATLAS-594: -- Summary: alter table rename doesnt work across databases Key: ATLAS-594 URL: https://issues.apache.org/jira/browse/ATLAS-594 Project: Atlas Issue Type: Sub-task Reporter: Suma Shivaprasad Alter table rename across dbs doesnt work – alter table db1.tb1 rename to db2.tb2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-588) import-hive.sh fails while importing partitions for a non-partitioned table
[
https://issues.apache.org/jira/browse/ATLAS-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad updated ATLAS-588:
---
Attachment: ATLAS-588.2.patch
Changed to StringUtils.join(hivePart.getValues(), ",") instead of Joiner since
that could lead to NPE
> import-hive.sh fails while importing partitions for a non-partitioned table
> ---
>
> Key: ATLAS-588
> URL: https://issues.apache.org/jira/browse/ATLAS-588
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-588.1.patch, ATLAS-588.2.patch, ATLAS-588.patch
>
>
> {noformat}
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]# export
> HIVE_CONF_DIR=/etc/hive/conf
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]#
> bin/import-hive.sh
> Using Hive configuration directory [/etc/hive/conf]
> Log file for import is /var/log/atlas/import-hive.log
> Exception in thread "main" org.codehaus.jettison.json.JSONException:
> JSONArray[0] not found.
> at org.codehaus.jettison.json.JSONArray.get(JSONArray.java:193)
> at org.codehaus.jettison.json.JSONArray.getString(JSONArray.java:316)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:191)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartition(HiveMetaStoreBridge.java:452)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartitions(HiveMetaStoreBridge.java:438)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importTables(HiveMetaStoreBridge.java:256)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:122)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:114)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:608)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (ATLAS-588) import-hive.sh fails while importing partitions for a non-partitioned table
[
https://issues.apache.org/jira/browse/ATLAS-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad updated ATLAS-588:
---
Attachment: ATLAS-588.1.patch
Fixed review comment
> import-hive.sh fails while importing partitions for a non-partitioned table
> ---
>
> Key: ATLAS-588
> URL: https://issues.apache.org/jira/browse/ATLAS-588
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-588.1.patch, ATLAS-588.patch
>
>
> {noformat}
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]# export
> HIVE_CONF_DIR=/etc/hive/conf
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]#
> bin/import-hive.sh
> Using Hive configuration directory [/etc/hive/conf]
> Log file for import is /var/log/atlas/import-hive.log
> Exception in thread "main" org.codehaus.jettison.json.JSONException:
> JSONArray[0] not found.
> at org.codehaus.jettison.json.JSONArray.get(JSONArray.java:193)
> at org.codehaus.jettison.json.JSONArray.getString(JSONArray.java:316)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:191)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartition(HiveMetaStoreBridge.java:452)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartitions(HiveMetaStoreBridge.java:438)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importTables(HiveMetaStoreBridge.java:256)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:122)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:114)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:608)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (ATLAS-588) import-hive.sh fails while importing partitions for a non-partitioned table
[
https://issues.apache.org/jira/browse/ATLAS-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad updated ATLAS-588:
---
Attachment: ATLAS-588.patch
> import-hive.sh fails while importing partitions for a non-partitioned table
> ---
>
> Key: ATLAS-588
> URL: https://issues.apache.org/jira/browse/ATLAS-588
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-588.patch
>
>
> {noformat}
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]# export
> HIVE_CONF_DIR=/etc/hive/conf
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]#
> bin/import-hive.sh
> Using Hive configuration directory [/etc/hive/conf]
> Log file for import is /var/log/atlas/import-hive.log
> Exception in thread "main" org.codehaus.jettison.json.JSONException:
> JSONArray[0] not found.
> at org.codehaus.jettison.json.JSONArray.get(JSONArray.java:193)
> at org.codehaus.jettison.json.JSONArray.getString(JSONArray.java:316)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:191)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartition(HiveMetaStoreBridge.java:452)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartitions(HiveMetaStoreBridge.java:438)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importTables(HiveMetaStoreBridge.java:256)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:122)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:114)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:608)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-588) import-hive.sh fails while importing partitions for a non-partitioned table
[ https://issues.apache.org/jira/browse/ATLAS-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209419#comment-15209419 ] Suma Shivaprasad commented on ATLAS-588: >From the table description, it clearly shows that the table is not partiotioned 0: jdbc:hive2://localhost:1> show create table table_1; +--+--+ | createtab_stmt | +--+--+ | CREATE TABLE `table_1`( | | `timestamp_col_1` timestamp, | | `decimal3003_col_2` decimal(30,3), | | `tinyint_col_3` tinyint, | | `decimal0101_col_4` decimal(1,1), | | `boolean_col_5` boolean, | | `float_col_6` float, | | `bigint_col_7` bigint, | | `varchar0098_col_8` varchar(98), | | `timestamp_col_9` timestamp, | | `bigint_col_10` bigint, | | `decimal0903_col_11` decimal(9,3), | | `timestamp_col_12` timestamp, | | `timestamp_col_13` timestamp, | | `float_col_14` float, | | `char0254_col_15` char(254), | | `double_col_16` double, | | `timestamp_col_17` timestamp, | | `boolean_col_18` boolean, | | `decimal2608_col_19` decimal(26,8), | | `varchar0216_col_20` varchar(216), | | `string_col_21` string, | | `bigint_col_22` bigint, | | `boolean_col_23` boolean, | | `timestamp_col_24` timestamp, | | `boolean_col_25` boolean, | | `decimal2016_col_26` decimal(20,16), | | `string_col_27` string, | | `decimal0202_col_28` decimal(2,2), | | `float_col_29` float, | | `decimal2020_col_30` decimal(20,20), | | `boolean_col_31` boolean, | | `double_col_32` double, | | `varchar0148_col_33` varchar(148), | | `decimal2121_col_34` decimal(21,21), | | `tinyint_col_35` tinyint,
[jira] [Comment Edited] (ATLAS-588) import-hive.sh fails while importing partitions for a non-partitioned table
[
https://issues.apache.org/jira/browse/ATLAS-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209413#comment-15209413
]
Suma Shivaprasad edited comment on ATLAS-588 at 3/23/16 11:32 PM:
--
When the bridge tries to import partitions for a specific table, the hive calls
to getPartitions is returning results even though the table is not partitioned
in the first place.
Eg : See table_1 in autoquerygen database which failed during the import.
List tableParts = hiveClient.getPartitions(table); should not return
any results but this seems to be the case and this is causing hive import to
fail.
"show partitions table_1" fails with the error
0: jdbc:hive2://localhost:1> show partitions table_1;
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLTask. Table table_1 is not a partitioned
table (state=08S01,code=1)
was (Author: suma.shivaprasad):
"show partitions table_1" fails with the error
0: jdbc:hive2://localhost:1> show partitions table_1;
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLTask. Table table_1 is not a partitioned
table (state=08S01,code=1)
> import-hive.sh fails while importing partitions for a non-partitioned table
> ---
>
> Key: ATLAS-588
> URL: https://issues.apache.org/jira/browse/ATLAS-588
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
>
> {noformat}
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]# export
> HIVE_CONF_DIR=/etc/hive/conf
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]#
> bin/import-hive.sh
> Using Hive configuration directory [/etc/hive/conf]
> Log file for import is /var/log/atlas/import-hive.log
> Exception in thread "main" org.codehaus.jettison.json.JSONException:
> JSONArray[0] not found.
> at org.codehaus.jettison.json.JSONArray.get(JSONArray.java:193)
> at org.codehaus.jettison.json.JSONArray.getString(JSONArray.java:316)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:191)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartition(HiveMetaStoreBridge.java:452)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartitions(HiveMetaStoreBridge.java:438)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importTables(HiveMetaStoreBridge.java:256)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:122)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:114)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:608)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-588) import-hive.sh fails while importing partitions for a non-partitioned table
[
https://issues.apache.org/jira/browse/ATLAS-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209417#comment-15209417
]
Suma Shivaprasad commented on ATLAS-588:
As can be seen in the above entity submission logs, the partitions keys for the
table are null. The workaround/fix for this is for ATLAS code to check if
partition.values is null , then do not proceed with adding the partitions.
> import-hive.sh fails while importing partitions for a non-partitioned table
> ---
>
> Key: ATLAS-588
> URL: https://issues.apache.org/jira/browse/ATLAS-588
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
>
> {noformat}
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]# export
> HIVE_CONF_DIR=/etc/hive/conf
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]#
> bin/import-hive.sh
> Using Hive configuration directory [/etc/hive/conf]
> Log file for import is /var/log/atlas/import-hive.log
> Exception in thread "main" org.codehaus.jettison.json.JSONException:
> JSONArray[0] not found.
> at org.codehaus.jettison.json.JSONArray.get(JSONArray.java:193)
> at org.codehaus.jettison.json.JSONArray.getString(JSONArray.java:316)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:191)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartition(HiveMetaStoreBridge.java:452)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartitions(HiveMetaStoreBridge.java:438)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importTables(HiveMetaStoreBridge.java:256)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:122)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:114)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:608)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-588) import-hive.sh fails while importing partitions for a non-partitioned table
[
https://issues.apache.org/jira/browse/ATLAS-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209415#comment-15209415
]
Suma Shivaprasad commented on ATLAS-588:
Summary of debugs from the log
{noformat}
2016-03-23 22:37:53,490 DEBUG - [main:] ~ Getting reference for partition for
autoquerygen.table_1 with values ['null'] (HiveMetaStoreBridge:391)
2016-03-23 22:37:54,214 DEBUG - [main:] ~ Submitting new entity hive_partition
= {
"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id":{
"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id":"-58877927966357",
"version":0,
"typeName":"hive_partition"
},
"typeName":"hive_partition",
"values":{
"createTime":0,
"qualifiedName":"autoquerygen.table_1.null@default",
"lastAccessTime":0,
"values":null,
"sd":{
"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id":{
"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id":"2c6f507c-6f71-4584-a889-ea3599240851",
"version":0,
"typeName":"hive_storagedesc"
},
"typeName":"hive_storagedesc",
"values":{
},
"traitNames":[
],
"traits":{
}
},
"parameters":null,
"table":{
"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id":{
"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id":"68a50aa5-4ffb-4f39-88f2-ba69c94c8b1e",
"version":0,
"typeName":"hive_table"
},
"typeName":"hive_table",
"values":{
"tableType":"MANAGED_TABLE",
"name":"autoquerygen.table_1@default",
"createTime":"1458726079",
"temporary":false,
"db":{
"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
{noformat}
> import-hive.sh fails while importing partitions for a non-partitioned table
> ---
>
> Key: ATLAS-588
> URL: https://issues.apache.org/jira/browse/ATLAS-588
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
>
> {noformat}
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]# export
> HIVE_CONF_DIR=/etc/hive/conf
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]#
> bin/import-hive.sh
> Using Hive configuration directory [/etc/hive/conf]
> Log file for import is /var/log/atlas/import-hive.log
> Exception in thread "main" org.codehaus.jettison.json.JSONException:
> JSONArray[0] not found.
> at org.codehaus.jettison.json.JSONArray.get(JSONArray.java:193)
> at org.codehaus.jettison.json.JSONArray.getString(JSONArray.java:316)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:191)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartition(HiveMetaStoreBridge.java:452)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartitions(HiveMetaStoreBridge.java:438)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importTables(HiveMetaStoreBridge.java:256)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:122)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:114)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:608)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-588) import-hive.sh fails while importing partitions for a non-partitioned table
[
https://issues.apache.org/jira/browse/ATLAS-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209413#comment-15209413
]
Suma Shivaprasad commented on ATLAS-588:
"show partitions table_1" fails with the error
0: jdbc:hive2://localhost:1> show partitions table_1;
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLTask. Table table_1 is not a partitioned
table (state=08S01,code=1)
> import-hive.sh fails while importing partitions for a non-partitioned table
> ---
>
> Key: ATLAS-588
> URL: https://issues.apache.org/jira/browse/ATLAS-588
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
>
> {noformat}
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]# export
> HIVE_CONF_DIR=/etc/hive/conf
> [root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]#
> bin/import-hive.sh
> Using Hive configuration directory [/etc/hive/conf]
> Log file for import is /var/log/atlas/import-hive.log
> Exception in thread "main" org.codehaus.jettison.json.JSONException:
> JSONArray[0] not found.
> at org.codehaus.jettison.json.JSONArray.get(JSONArray.java:193)
> at org.codehaus.jettison.json.JSONArray.getString(JSONArray.java:316)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:191)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartition(HiveMetaStoreBridge.java:452)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartitions(HiveMetaStoreBridge.java:438)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importTables(HiveMetaStoreBridge.java:256)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:122)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:114)
> at
> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:608)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (ATLAS-588) import-hive.sh fails while importing partitions for a non-partitioned table
Suma Shivaprasad created ATLAS-588:
--
Summary: import-hive.sh fails while importing partitions for a
non-partitioned table
Key: ATLAS-588
URL: https://issues.apache.org/jira/browse/ATLAS-588
Project: Atlas
Issue Type: Bug
Affects Versions: 0.7-incubating
Reporter: Suma Shivaprasad
Assignee: Suma Shivaprasad
Fix For: 0.7-incubating
{noformat}
[root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]# export
HIVE_CONF_DIR=/etc/hive/conf
[root@sneethiraj-ambari-atlas-job-160323-0215-1 atlas-server]#
bin/import-hive.sh
Using Hive configuration directory [/etc/hive/conf]
Log file for import is /var/log/atlas/import-hive.log
Exception in thread "main" org.codehaus.jettison.json.JSONException:
JSONArray[0] not found.
at org.codehaus.jettison.json.JSONArray.get(JSONArray.java:193)
at org.codehaus.jettison.json.JSONArray.getString(JSONArray.java:316)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:191)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartition(HiveMetaStoreBridge.java:452)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerPartitions(HiveMetaStoreBridge.java:438)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importTables(HiveMetaStoreBridge.java:256)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:122)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:114)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:608)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (ATLAS-535) Support delete cascade efficiently
[ https://issues.apache.org/jira/browse/ATLAS-535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-535: --- Summary: Support delete cascade efficiently (was: Support delete cascade efficently) > Support delete cascade efficiently > -- > > Key: ATLAS-535 > URL: https://issues.apache.org/jira/browse/ATLAS-535 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > > Currently there are some limitation in the typesystem and modelling to > support delete cascades at scale through the isComposite flag -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-553) Entity mutation - Fix issue with reordering of elements in array with composite references
[ https://issues.apache.org/jira/browse/ATLAS-553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-553: --- Summary: Entity mutation - Fix issue with reordering of elements in array with composite references (was: Entity mutation - Fix issue with reordering of elements in array) > Entity mutation - Fix issue with reordering of elements in array with > composite references > - > > Key: ATLAS-553 > URL: https://issues.apache.org/jira/browse/ATLAS-553 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-553.1.patch, ATLAS-553.patch > > > Currently, if theres an update on the entity to the order of elements in an > array/aray etc, it is not handled correctly since entity > mutation logic thinks that the element is deleted resulting in a deleted > vertex being referred later causing entity mutation to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-528: --- Description: Drop table and view requires soft deletes. The reason is whenever a table is dropped , it also may have an associated lineage which consists of a hive_process which N input tables and an output table. If the table is dropped, the lineage edge is also dropped resulting in incorrect lineage history. With soft deletes, the expected behaviour is to changes the table status to "deleted" and when the table is recreated through a create table statement, then create another vertex/entity for that table with the new state. Also, the lineage for this newly recreated table should be a new hive_process and should not reuse the existing entity/vertex even though the hive statement for that process is the same. was:Drop table and view requires soft deletes. The reason is whenever a table is dropped , it also may have an associated lineage which consists of a hive_process which N input tables and an output table. If the table is dropped, the lineage edge is also dropped resulting in incorrect lineage history > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-528: --- Description: Drop table and view requires soft deletes. The reason is whenever a table is dropped , it also may have an associated lineage which consists of a hive_process which N input tables and an output table. If the table is dropped, the lineage edge is also dropped resulting in incorrect lineage history > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (ATLAS-534) Verify free text search is able to surface search on table parameters
[ https://issues.apache.org/jira/browse/ATLAS-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad resolved ATLAS-534. Resolution: Won't Fix Assignee: Suma Shivaprasad Was able to verify that free text search is able to surface any parameters stored in the table/db parameters > Verify free text search is able to surface search on table parameters > - > > Key: ATLAS-534 > URL: https://issues.apache.org/jira/browse/ATLAS-534 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ATLAS-583) Rename table should retain traits/tags assigned to columns
Suma Shivaprasad created ATLAS-583: -- Summary: Rename table should retain traits/tags assigned to columns Key: ATLAS-583 URL: https://issues.apache.org/jira/browse/ATLAS-583 Project: Atlas Issue Type: Sub-task Reporter: Suma Shivaprasad Currently rename changes the column qualifiedName in the renamed table resulting in the columns getting deleted and therefore the traits attached to these will also get deleted. We need to ensure they are retained by ensuring columns are renamed first to the new qualified name and then table is renamed in the repository -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (ATLAS-531) Remove isComposite on storageDescriptor.cols
[ https://issues.apache.org/jira/browse/ATLAS-531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad resolved ATLAS-531. Resolution: Fixed Assignee: Suma Shivaprasad This is not required currently since the concerned attribute has been removed from the model > Remove isComposite on storageDescriptor.cols > > > Key: ATLAS-531 > URL: https://issues.apache.org/jira/browse/ATLAS-531 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > > This is causing issues since Table.cols and storageDesc.cols have isComposite > attributes and try to control the lifecycle causing issues -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-568) Parallelize Hive hook operations
[ https://issues.apache.org/jira/browse/ATLAS-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-568: --- Description: Maintaining the same order of operations that were executed in hive is crucial also on top of ATLAS . This is because if they are not ordered, it could easily lead to correctness issues in the ATLAS repository. For eg: Table columns being dropped and then table is renamed , dropping tables, databases etc all need to be executed in the same order as they were in hive metastore. There are multiple issues that needs to be addressed here 1. How do we ensure order of messages on the producer/hook side? 2. Once producer/hook publishes these messages onto KAFKA, how do we ensure the order of processing is the same as it was published. One suggested approach is to assign a timestamp to all the messages on the producer side and have a window/batch these messages on the consumer/ATLAS server side. was: Maintaining the same order of operations that were executed in hive is crucial also on top of ATLAS . This is because if they are not ordered, it could easily lead to correctness issues in the ATLAS repository. For eg: Table columns being dropped and then table is renamed , dropping tables, databases etc all need to be executed in the same order as they were in hive metastore. There are multiple issues that needs to be addressed here 1. How do we ensure order of messages on the producer/hook side? 2. Once producer/hook publishes these messages onto KAFKA, how do we ensure the order of processing is the same as it was published. One suggested approach is to assign a timestamp to all the messages on the producer side and have a window/batch these messages on the consumer/ATLAS server side. Order these messages according to the timestamp within the window which is a configured time period and then execute these operations in that order. > Parallelize Hive hook operations > - > > Key: ATLAS-568 > URL: https://issues.apache.org/jira/browse/ATLAS-568 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > > Maintaining the same order of operations that were executed in hive is > crucial also on top of ATLAS . This is because if they are not ordered, it > could easily lead to correctness issues in the ATLAS repository. For eg: > Table columns being dropped and then table is renamed , dropping tables, > databases etc all need to be executed in the same order as they were in hive > metastore. There are multiple issues that needs to be addressed here > 1. How do we ensure order of messages on the producer/hook side? > 2. Once producer/hook publishes these messages onto KAFKA, how do we ensure > the order of processing is the same as it was published. > One suggested approach is to assign a timestamp to all the messages on the > producer side and have a window/batch these messages on the consumer/ATLAS > server side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-568) Parallelize Hive hook operations
[ https://issues.apache.org/jira/browse/ATLAS-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-568: --- Description: Maintaining the same order of operations that were executed in hive is crucial also on top of ATLAS . This is because if they are not ordered, it could easily lead to correctness issues in the ATLAS repository. For eg: Table columns being dropped and then table is renamed , dropping tables, databases etc all need to be executed in the same order as they were in hive metastore. There are multiple issues that needs to be addressed here 1. How do we ensure order of messages on the producer/hook side? 2. Once producer/hook publishes these messages onto KAFKA, how do we ensure the order of processing is the same as it was published. One suggested approach is to assign a timestamp to all the messages on the producer side and have a window/batch these messages on the consumer/ATLAS server side. Order these messages according to the timestamp within the window which is a configured time period and then execute these operations in that order. was:Currently, there are operations like rename which fire multiple messages to repository to accomplish the required task(create table by old name if it doesnt exist. This could lead to issues if there are multiple consumers running as kafka consumers since the order of messages could be executed in anyway in this case resulting in inconsistent behaviour. > Parallelize Hive hook operations > - > > Key: ATLAS-568 > URL: https://issues.apache.org/jira/browse/ATLAS-568 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > > Maintaining the same order of operations that were executed in hive is > crucial also on top of ATLAS . This is because if they are not ordered, it > could easily lead to correctness issues in the ATLAS repository. For eg: > Table columns being dropped and then table is renamed , dropping tables, > databases etc all need to be executed in the same order as they were in hive > metastore. There are multiple issues that needs to be addressed here > 1. How do we ensure order of messages on the producer/hook side? > 2. Once producer/hook publishes these messages onto KAFKA, how do we ensure > the order of processing is the same as it was published. > One suggested approach is to assign a timestamp to all the messages on the > producer side and have a window/batch these messages on the consumer/ATLAS > server side. Order these messages according to the timestamp within the > window which is a configured time period and then execute these operations in > that order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ATLAS-568) Parallelize Hive hook operations
[ https://issues.apache.org/jira/browse/ATLAS-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-568: --- Summary: Parallelize Hive hook operations (was: Make hive hook multithreaded) > Parallelize Hive hook operations > - > > Key: ATLAS-568 > URL: https://issues.apache.org/jira/browse/ATLAS-568 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad > Fix For: 0.7-incubating > > > Currently, there are operations like rename which fire multiple messages to > repository to accomplish the required task(create table by old name if it > doesnt exist. This could lead to issues if there are multiple consumers > running as kafka consumers since the order of messages could be executed in > anyway in this case resulting in inconsistent behaviour. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-575) jetty-maven-plugin fails with ShutdownMonitorThread already started
[
https://issues.apache.org/jira/browse/ATLAS-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206869#comment-15206869
]
Suma Shivaprasad commented on ATLAS-575:
+1
> jetty-maven-plugin fails with ShutdownMonitorThread already started
> ---
>
> Key: ATLAS-575
> URL: https://issues.apache.org/jira/browse/ATLAS-575
> Project: Atlas
> Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Shwetha G S
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-575-v2.patch, ATLAS-575.patch
>
>
> {noformat}
> [INFO] --- jetty-maven-plugin:9.2.12.v20150709:deploy-war (start-jetty) @
> falcon-bridge ---
> [INFO] Configuring Jetty for project: Apache Atlas Falcon Bridge
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Atlas UI SUCCESS [01:04
> min]
> [INFO] Apache Atlas Web Application ... SUCCESS [03:13
> min]
> [INFO] Apache Atlas Documentation . SUCCESS [ 4.433
> s]
> [INFO] Apache Atlas Hive Bridge ... SUCCESS [01:31
> min]
> [INFO] Apache Atlas Falcon Bridge . FAILURE [ 3.805
> s]
> [INFO] Apache Atlas Sqoop Bridge .. SKIPPED
> [INFO] Apache Atlas Storm Bridge .. SKIPPED
> [INFO] Apache Atlas Distribution .. SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 05:58 min
> [INFO] Finished at: 2016-03-18T12:42:32+05:30
> [INFO] Final Memory: 188M/687M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.eclipse.jetty:jetty-maven-plugin:9.2.12.v20150709:deploy-war
> (start-jetty) on project falcon-bridge: Failure: ShutdownMonitorThread
> already started -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (ATLAS-553) Entity mutation - Fix issue with reordering of elements in array
[ https://issues.apache.org/jira/browse/ATLAS-553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated ATLAS-553: --- Attachment: ATLAS-553.1.patch Attaching patch after fixing [~shwethags] review comments which was to change the globalIdToVertexMap to a set. > Entity mutation - Fix issue with reordering of elements in array > -- > > Key: ATLAS-553 > URL: https://issues.apache.org/jira/browse/ATLAS-553 > Project: Atlas > Issue Type: Bug >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-553.1.patch, ATLAS-553.patch > > > Currently, if theres an update on the entity to the order of elements in an > array/aray etc, it is not handled correctly since entity > mutation logic thinks that the element is deleted resulting in a deleted > vertex being referred later causing entity mutation to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
