[jira] [Commented] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233190#comment-15233190 ] Hemanth Yamijala commented on ATLAS-528: +1 for ATLAS_528.2.patch > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.1.patch, ATLAS-528.patch, ATLAS_528.2.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[
https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232662#comment-15232662
]
ATLAS QA commented on ATLAS-528:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12797755/ATLAS_528.2.patch
against master revision a908001.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
+1 checkstyle. The patch generated 0 code style errors.
{color:red}-1 findbugs{color}. The patch appears to introduce 352 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryTest
./repository/target/surefire-reports/junitreports/TEST-org.apache.atlas.repository.audit.HBaseBasedAuditRepositoryTest
Test results:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningsrepository.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningstitan.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningswebapp.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningstypesystem.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningsclient.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningsnotification.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningscommon.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningshive-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningshdfs-model.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningsstorm-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/142//artifact/patchprocess/newPatchFindbugsWarningssqoop-bridge.html
Console output: https://builds.apache.org/job/PreCommit-ATLAS-Build/142//console
This message is automatically generated.
> support drop table, view
>
>
> Key: ATLAS-528
> URL: https://issues.apache.org/jira/browse/ATLAS-528
> Project: Atlas
> Issue Type: Sub-task
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-528.1.patch, ATLAS-528.patch, ATLAS_528.2.patch
>
>
> Drop table and view requires soft deletes. The reason is whenever a table is
> dropped , it also may have an associated lineage which consists of a
> hive_process which N input tables and an output table. If the table is
> dropped, the lineage edge is also dropped resulting in incorrect lineage
> history.
> With soft deletes, the expected behaviour is to changes the table status to
> "deleted" and when the table is recreated through a create table statement,
> then create another vertex/entity for that table with the new state. Also,
> the lineage for this newly recreated table should be a new hive_process and
> should not reuse the existing entity/vertex even though the hive statement
> for that process is the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232501#comment-15232501 ] Suma Shivaprasad commented on ATLAS-528: Had intended to remove DROPVIEW_PROPERTIES earlier since I had noticed that the hive parser doesnt result in this HIveOperation enum type in any way while trying to come up with a test case for it . It is an extra unused HiveOperation in hive code. Had missed removing it. Thanks for catching this. Will upload the patch again after removing this > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.1.patch, ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231788#comment-15231788 ] Hemanth Yamijala commented on ATLAS-528: Changes look OK to me. Just one minor question. This patch adds a DROPVIEW_PROPERTIES to the list of handled commands. I did not understand what this command does? I tried executing something like DROP VIEW view_name PROPERTIES etc, and wasn't successful in getting the right command. Can you please explain what it does? If you're sure about how we've handled DROPVIEW_PROPERTIES, then please go ahead with the commit. > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.1.patch, ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[
https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231076#comment-15231076
]
ATLAS QA commented on ATLAS-528:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12797593/ATLAS-528.1.patch
against master revision 46365f8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
+1 checkstyle. The patch generated 0 code style errors.
{color:red}-1 findbugs{color}. The patch appears to introduce 352 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
./webapp/test-output/junitreports/TEST-org.apache.atlas.web.resources.RexsterGraphJerseyResourceIT
./webapp/test-output/junitreports/TEST-org.apache.atlas.web.resources.AdminJerseyResourceIT
./webapp/test-output/junitreports/TEST-org.apache.atlas.web.resources.EntityJerseyResourceIT
./webapp/test-output/junitreports/TEST-org.apache.atlas.web.resources.MetadataDiscoveryJerseyResourceIT
./webapp/test-output/junitreports/TEST-org.apache.atlas.web.resources.TypesJerseyResourceIT
Test results:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningswebapp.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningscommon.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningssqoop-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningshdfs-model.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningsstorm-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningshive-bridge.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningsrepository.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningstypesystem.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningsclient.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningsnotification.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-ATLAS-Build/139//artifact/patchprocess/newPatchFindbugsWarningstitan.html
Console output: https://builds.apache.org/job/PreCommit-ATLAS-Build/139//console
This message is automatically generated.
> support drop table, view
>
>
> Key: ATLAS-528
> URL: https://issues.apache.org/jira/browse/ATLAS-528
> Project: Atlas
> Issue Type: Sub-task
>Affects Versions: 0.7-incubating
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-528.1.patch, ATLAS-528.patch
>
>
> Drop table and view requires soft deletes. The reason is whenever a table is
> dropped , it also may have an associated lineage which consists of a
> hive_process which N input tables and an output table. If the table is
> dropped, the lineage edge is also dropped resulting in incorrect lineage
> history.
> With soft deletes, the expected behaviour is to changes the table status to
> "deleted" and when the table is recreated through a create table statement,
> then create another vertex/entity for that table with the new state. Also,
> the lineage for this newly recreated table should be a new hive_process and
> should not reuse the existing entity/vertex even though the hive statement
> for that process is the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230983#comment-15230983 ] Suma Shivaprasad commented on ATLAS-528: @Hemanth yamijala Thanks for the review. The handling of truncate table will create a single process with no inputs and a single table as output. We should validate if this type of lineage view is OK for a end user. I agree though that capturing the information is important. --> Raised ATLAS-653 to track this. On the same note, is the query string captured correctly for the process. Can we enhance the truncate test to validate this? --> HiveHookIT.validateProcess already check this by querying with the same exact query If a table / view being dropped is not present in Atlas, this would generate a 404 and we would loose the information. Maybe we can file a bug and track this for later? --> Raised ATLAS-652 to track this. We should also test what happens if a user executes a drop table if exists non_existent_table - do we get called? --> I observed that the hooks is being called and the hook is ignoring this since there are no outputs for this query. Added a testcase HiveHookIT.testDropNonExistingTable to test this. There are break statements missing after ALTERTABLE_LOCATION and DROPTABLE/DROPVIEW. While the latter are the last branches of the switch, it is still safer to add IMO. --> Added > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Affects Versions: 0.7-incubating >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.1.patch, ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229913#comment-15229913 ] Hemanth Yamijala commented on ATLAS-528: Talking to [~shwethags], it appears the lineage query in the backend for a particular dataset (say inputs for this dataset), will not return just the process information if there is no corresponding dataset that is created as output of this process. So, maybe the truncate is not going to work as I have mentioned. Can you please verify? > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[
https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229797#comment-15229797
]
Hemanth Yamijala commented on ATLAS-528:
Applied the patch on top of ATLAS-527 and reviewed it. A few comments:
* The handling of truncate table will create a single process with no inputs
and a single table as output. We should validate if this type of lineage view
is OK for a end user. I agree though that capturing the information is
important.
* On the same note, is the query string captured correctly for the process. Can
we enhance the truncate test to validate this?
* If a table / view being dropped is not present in Atlas, this would generate
a 404 and we would loose the information. Maybe we can file a bug and track
this for later?
* We should also test what happens if a user executes a {{drop table if exists
non_existent_table}} - do we get called?
* There are break statements missing after {{ALTERTABLE_LOCATION}} and
{{DROPTABLE/DROPVIEW}}. While the latter are the last branches of the switch,
it is still safer to add IMO.
> support drop table, view
>
>
> Key: ATLAS-528
> URL: https://issues.apache.org/jira/browse/ATLAS-528
> Project: Atlas
> Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-528.patch
>
>
> Drop table and view requires soft deletes. The reason is whenever a table is
> dropped , it also may have an associated lineage which consists of a
> hive_process which N input tables and an output table. If the table is
> dropped, the lineage edge is also dropped resulting in incorrect lineage
> history.
> With soft deletes, the expected behaviour is to changes the table status to
> "deleted" and when the table is recreated through a create table statement,
> then create another vertex/entity for that table with the new state. Also,
> the lineage for this newly recreated table should be a new hive_process and
> should not reuse the existing entity/vertex even though the hive statement
> for that process is the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229324#comment-15229324 ] Suma Shivaprasad commented on ATLAS-528: Also have added support for truncate table, alter table partition column type as part of the same patch > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-528) support drop table, view
[ https://issues.apache.org/jira/browse/ATLAS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229315#comment-15229315 ] Suma Shivaprasad commented on ATLAS-528: This is dependent on patch of ATLAS-527 which is under review. Hence not making it Patch Available. > support drop table, view > > > Key: ATLAS-528 > URL: https://issues.apache.org/jira/browse/ATLAS-528 > Project: Atlas > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.7-incubating > > Attachments: ATLAS-528.patch > > > Drop table and view requires soft deletes. The reason is whenever a table is > dropped , it also may have an associated lineage which consists of a > hive_process which N input tables and an output table. If the table is > dropped, the lineage edge is also dropped resulting in incorrect lineage > history. > With soft deletes, the expected behaviour is to changes the table status to > "deleted" and when the table is recreated through a create table statement, > then create another vertex/entity for that table with the new state. Also, > the lineage for this newly recreated table should be a new hive_process and > should not reuse the existing entity/vertex even though the hive statement > for that process is the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
