[jira] [Commented] (HIVE-13112) Expose Lineage information in case of CTAS
[ https://issues.apache.org/jira/browse/HIVE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15469722#comment-15469722 ] Vimal Sharma commented on HIVE-13112: - [~rhbutani] Can you please verify the issue and help. I have created a JIRA for this issue at https://issues.apache.org/jira/browse/HIVE-14706 > Expose Lineage information in case of CTAS > -- > > Key: HIVE-13112 > URL: https://issues.apache.org/jira/browse/HIVE-13112 > Project: Hive > Issue Type: Bug > Components: lineage >Reporter: Harish Butani >Assignee: Harish Butani > Fix For: 2.1.0 > > Attachments: HIVE-13112.01.patch > > > This is not happening because Lineage is being exposed by MoveTask which > checks for a LoadTableDesc. In case of CTAS the table is created after the > MoveTask. > Proposed solution is to add a flag to CreateTableDesc to track a CTAS > operation, and have the DDLTask expose the lineage if this flag is set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13112) Expose Lineage information in case of CTAS
[
https://issues.apache.org/jira/browse/HIVE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15458509#comment-15458509
]
Vimal Sharma commented on HIVE-13112:
-
I am using Hive version 2.1.0 and trying to get Column level lineage. However,
when I try to access the Dependency value in LineageInfo, it is coming as null.
Sample Code:
{code}
public static Map>
buildLineageMap(LineageInfo lInfo) {
for(Map.Entry e :
lInfo.entrySet()) {
System.out.println("Col Lineage Key : " + e.getKey());
System.out.println("Col Lineage Value: " + e.getValue());
{code}
In the output, Key is set to destination column in a lineage but the Value
which should contain origin column information is null. How can I verify
whether the lineage information being set correctly in Hive Sessionstate.
> Expose Lineage information in case of CTAS
> --
>
> Key: HIVE-13112
> URL: https://issues.apache.org/jira/browse/HIVE-13112
> Project: Hive
> Issue Type: Bug
> Components: lineage
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 2.1.0
>
> Attachments: HIVE-13112.01.patch
>
>
> This is not happening because Lineage is being exposed by MoveTask which
> checks for a LoadTableDesc. In case of CTAS the table is created after the
> MoveTask.
> Proposed solution is to add a flag to CreateTableDesc to track a CTAS
> operation, and have the DDLTask expose the lineage if this flag is set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13112) Expose Lineage information in case of CTAS
[ https://issues.apache.org/jira/browse/HIVE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186439#comment-15186439 ] Ashutosh Chauhan commented on HIVE-13112: - +1 > Expose Lineage information in case of CTAS > -- > > Key: HIVE-13112 > URL: https://issues.apache.org/jira/browse/HIVE-13112 > Project: Hive > Issue Type: Bug >Reporter: Harish Butani >Assignee: Harish Butani > Attachments: HIVE-13112.01.patch > > > This is not happening because Lineage is being exposed by MoveTask which > checks for a LoadTableDesc. In case of CTAS the table is created after the > MoveTask. > Proposed solution is to add a flag to CreateTableDesc to track a CTAS > operation, and have the DDLTask expose the lineage if this flag is set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13112) Expose Lineage information in case of CTAS
[
https://issues.apache.org/jira/browse/HIVE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179392#comment-15179392
]
Hive QA commented on HIVE-13112:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789075/HIVE-13112.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 226 failed/errored test(s), 9781 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition_authorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_update_status
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_create_temp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_view_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_view_disable_cbo_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autogen_colalias
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_SortUnionTransposeRule
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_unionDistinct_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_nested_types
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_colstats_all_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_default_prop
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_char
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_colname
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_uses_database_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_varchar
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dbtxnmgr_ddl1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_empty_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_ddl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_global_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_duplicate_key
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input46
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join41
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_uncompressed
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_lateral_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_union_src
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nestedvirtual
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap
org.apache.hadoop.hiv
[jira] [Commented] (HIVE-13112) Expose Lineage information in case of CTAS
[ https://issues.apache.org/jira/browse/HIVE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157484#comment-15157484 ] Harish Butani commented on HIVE-13112: -- The HookContext gets the Lineage from the SessionState. And AFAICT the Lineage in SessionState is only being set in the MoveTask. > Expose Lineage information in case of CTAS > -- > > Key: HIVE-13112 > URL: https://issues.apache.org/jira/browse/HIVE-13112 > Project: Hive > Issue Type: Bug >Reporter: Harish Butani > > This is not happening because Lineage is being exposed by MoveTask which > checks for a LoadTableDesc. In case of CTAS the table is created after the > MoveTask. > Proposed solution is to add a flag to CreateTableDesc to track a CTAS > operation, and have the DDLTask expose the lineage if this flag is set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13112) Expose Lineage information in case of CTAS
[ https://issues.apache.org/jira/browse/HIVE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157399#comment-15157399 ] Jimmy Xiang commented on HIVE-13112: Have you tried the lineage hook: org.apache.hadoop.hive.ql.hooks.LineageLogger? > Expose Lineage information in case of CTAS > -- > > Key: HIVE-13112 > URL: https://issues.apache.org/jira/browse/HIVE-13112 > Project: Hive > Issue Type: Bug >Reporter: Harish Butani > > This is not happening because Lineage is being exposed by MoveTask which > checks for a LoadTableDesc. In case of CTAS the table is created after the > MoveTask. > Proposed solution is to add a flag to CreateTableDesc to track a CTAS > operation, and have the DDLTask expose the lineage if this flag is set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
