[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255976#comment-15255976 ] Nemon Lou commented on HIVE-13602: -- I just copy it from the result of "explain rewrite q16". {noformat} explain rewrite select p_brand, p_type, p_size, count(distinct ps_suppkey) as supplier_cnt from partsupp, part where p_partkey = ps_partkey and p_brand <> 'Brand#34' and p_type not like 'ECONOMY BRUSHED%' and p_size in (22, 14, 27, 49, 21, 33, 35, 28) and partsupp.ps_suppkey not in ( select s_suppkey from supplier where s_comment like '%Customer%Complaints%' ) group by p_brand, p_type, p_size order by supplier_cnt desc, p_brand, p_type, p_size; {noformat} > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255974#comment-15255974 ] Prasanth Jayachandran commented on HIVE-13536: -- [~sseth] Could you please take another look? > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255973#comment-15255973 ] Prasanth Jayachandran commented on HIVE-13097: -- LGMT, +1 > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.01.patch, HIVE-13097.02.patch, > HIVE-13097.03.patch, HIVE-13097.versonChange.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255964#comment-15255964 ] Nemon Lou commented on HIVE-13602: -- >From calcite rel plan ,we can see the p_brand column is declared to fetch. And the value for p_brand column is zero in the final result. {noformat} 0: jdbc:hive2://189.39.151.141:21066/> select * from q16_cbo_bad limit 1; ++---+---+-+--+ | q16_cbo_bad.p_brand | q16_cbo_bad.p_type | q16_cbo_bad.p_size | q16_cbo_bad.supplier_cnt | ++---+---+-+--+ | 0 | MEDIUM BRUSHED TIN| 21| 298 | ++---+---+-+--+ 1 row selected (0.139 seconds) 0: jdbc:hive2://189.39.151.141:21066/> select * from q16_cbo_rewrite limit 1; ++---+---+-+--+ | q16_cbo_rewrite.p_brand | q16_cbo_rewrite.p_type | q16_cbo_rewrite.p_size | q16_cbo_rewrite.supplier_cnt | ++---+---+-+--+ | Brand#31 | PROMO ANODIZED NICKEL | 14 | 40 | ++---+---+-+--+ 1 row selected (0.961 seconds) {noformat} > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255953#comment-15255953 ] Pengcheng Xiong commented on HIVE-13602: what is {code} 1 = 1 {code} in the where condition? > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255950#comment-15255950 ] Pengcheng Xiong commented on HIVE-13602: [~nemon], if p_brand is missing, i would expect to see that one column is missing, rather than different number of rows? > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255937#comment-15255937 ] Nemon Lou commented on HIVE-13602: -- The join order seems to be the same for both queries.The difference is that the bad plan only select three columns for table part at stage-1 (p_brand is missing),the good one select four. > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255934#comment-15255934 ] Hive QA commented on HIVE-13563: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800332/HIVE-13563.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9992 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/70/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/70/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-70/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12800332 - PreCommit-HIVE-MASTER-Build > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-13602: - Attachment: explain_cbo_good(rewrite)_part3.out explain_cbo_good(rewrite)_part2.out explain_cbo_good(rewrite)_part1.out calcite_cbo_good.out calcite_cbo_bad.out > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13603) Fix ptest unit tests broken by HIVE13505
[ https://issues.apache.org/jira/browse/HIVE-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13603: -- Status: Patch Available (was: Open) > Fix ptest unit tests broken by HIVE13505 > > > Key: HIVE-13603 > URL: https://issues.apache.org/jira/browse/HIVE-13603 > Project: Hive > Issue Type: Task >Affects Versions: 2.1.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13603.01.patch > > > HIVE-13505 broke some unit tests in the ptest2 framework, which need to be > fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13603) Fix ptest unit tests broken by HIVE13505
[ https://issues.apache.org/jira/browse/HIVE-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13603: -- Attachment: HIVE-13603.01.patch Simple fix. cc [~szehon], [~spena], [~ashutoshc] for review. Don't think this needs to take up a slot in the build queue. Submitting anyway for now. Will commit and cancel the precommit run if people are ok with that. There's still some failing tests after this, but those exist without HIVE-13505 as well. I suspect that has something to do with enabling / disabling the spark profile. > Fix ptest unit tests broken by HIVE13505 > > > Key: HIVE-13603 > URL: https://issues.apache.org/jira/browse/HIVE-13603 > Project: Hive > Issue Type: Task >Affects Versions: 2.1.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13603.01.patch > > > HIVE-13505 broke some unit tests in the ptest2 framework, which need to be > fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255921#comment-15255921 ] Pengcheng Xiong commented on HIVE-13602: [~nemon], could u also post the explain plan for the rewrite query? As I can see from the attachement, the original plan when CBO is on is to {code} (join part with supplier cnt=0) join (left outer join partsupp with supplier). {code} I would like to see if the rewrite query follows the same plan. > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255914#comment-15255914 ] Nemon Lou commented on HIVE-13602: -- Another information : after rewrite q16 into the following form,CBO can return right result: {noformat} select p_brand, p_type, p_size, count(distinct ps_suppkey) as supplier_cnt from partsupp left outer join ( select s_suppkey from supplier where s_comment like '%Customer%Complaints%' ) sq_1 on partsupp.ps_suppkey = sq_1.s_suppkey join ( select count(*) as cn from (select * from supplier where s_comment like '%Customer%Complaints%' ) b where s_suppkey is null) a on a.cn= 0 join part where p_partkey = ps_partkey and p_brand <> 'Brand#34' and p_type not like 'ECONOMY BRUSHED%' and p_size in (22, 14, 27, 49, 21, 33, 35, 28) and 1 = 1 and sq_1.s_suppkey is null group by p_brand, p_type, p_size order by supplier_cnt desc, p_brand, p_type, p_size; {noformat} > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13568) Add UDFs to support column-masking
[ https://issues.apache.org/jira/browse/HIVE-13568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13568: Attachment: HIVE-13568.1.patch > Add UDFs to support column-masking > -- > > Key: HIVE-13568 > URL: https://issues.apache.org/jira/browse/HIVE-13568 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj > Attachments: HIVE-13568.1.patch, HIVE-13568.1.patch > > > HIVE-13125 added support to provide column-masking and row-filtering during > select via HiveAuthorizer interface. This JIRA is track addition of UDFs that > can be used by HiveAuthorizer implementations to mask column values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13568) Add UDFs to support column-masking
[ https://issues.apache.org/jira/browse/HIVE-13568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13568: Status: Patch Available (was: Open) > Add UDFs to support column-masking > -- > > Key: HIVE-13568 > URL: https://issues.apache.org/jira/browse/HIVE-13568 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj > Attachments: HIVE-13568.1.patch, HIVE-13568.1.patch > > > HIVE-13125 added support to provide column-masking and row-filtering during > select via HiveAuthorizer interface. This JIRA is track addition of UDFs that > can be used by HiveAuthorizer implementations to mask column values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12959) LLAP: Add task scheduler timeout when no nodes are alive
[ https://issues.apache.org/jira/browse/HIVE-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12959: -- Attachment: HIVE-12959.7.patch Re-uploading for a jenkins run. > LLAP: Add task scheduler timeout when no nodes are alive > > > Key: HIVE-12959 > URL: https://issues.apache.org/jira/browse/HIVE-12959 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12959.1.patch, HIVE-12959.2.patch, > HIVE-12959.3.patch, HIVE-12959.5.patch, HIVE-12959.6.patch, > HIVE-12959.6.patch, HIVE-12959.7.patch, HIVE-12959.7.patch, HIVE-12959.7.txt > > > When there are no llap daemons running task scheduler should have a timeout > to fail the query instead of waiting forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13568) Add UDFs to support column-masking
[ https://issues.apache.org/jira/browse/HIVE-13568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13568: Status: Open (was: Patch Available) > Add UDFs to support column-masking > -- > > Key: HIVE-13568 > URL: https://issues.apache.org/jira/browse/HIVE-13568 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj > Attachments: HIVE-13568.1.patch > > > HIVE-13125 added support to provide column-masking and row-filtering during > select via HiveAuthorizer interface. This JIRA is track addition of UDFs that > can be used by HiveAuthorizer implementations to mask column values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255910#comment-15255910 ] Siddharth Seth commented on HIVE-13097: --- Test failures are unrelated. > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.01.patch, HIVE-13097.02.patch, > HIVE-13097.03.patch, HIVE-13097.versonChange.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13537) Update slf4j version to 1.7.10
[ https://issues.apache.org/jira/browse/HIVE-13537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13537: -- Attachment: HIVE-13537.01.patch > Update slf4j version to 1.7.10 > -- > > Key: HIVE-13537 > URL: https://issues.apache.org/jira/browse/HIVE-13537 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Minor > Attachments: HIVE-13537.01.patch, HIVE-13537.01.patch > > > Both Hadoop and Tez are on 1.7.10, and have been for a while. We should > update hive to use this version as well. Should get rid of some of the noise > in the logs related to multiple version.s -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-13602: - Attachment: explain_cbo_bad_part3.out explain_cbo_bad_part2.out explain_cbo_bad_part1.out I have run "analyze table .. for columns" for all the tables. Splitting the explain file into 3 parts in order to upload. > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13469) LLAP: Support delayed scheduling for locality
[ https://issues.apache.org/jira/browse/HIVE-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13469: -- Attachment: HIVE-13469.02.patch Re-uploading for precommit. > LLAP: Support delayed scheduling for locality > - > > Key: HIVE-13469 > URL: https://issues.apache.org/jira/browse/HIVE-13469 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13469.01.patch, HIVE-13469.02.patch, > HIVE-13469.02.patch > > > LLAP currently supports forcing locality. Change this to support a time based > delay for locality as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255900#comment-15255900 ] Pengcheng Xiong commented on HIVE-13602: [~nemon], do you have all the table/column stats for all the tables? Could u attach the explain results for CBO on and off? Thanks. > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255897#comment-15255897 ] Nemon Lou commented on HIVE-13602: -- I used MR engine. The attachment is bigger than 50K ,and has been rejected by firewall or something else. > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-13602: -- Assignee: Pengcheng Xiong > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255892#comment-15255892 ] Pengcheng Xiong commented on HIVE-13602: cc'ing [~ashutoshc] and [~hagleitn]. [~nemon], could u be more specific about the execution engine that you used? MR or Tez? > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12837) Better memory estimation/allocation for hybrid grace hash join during hash table loading
[ https://issues.apache.org/jira/browse/HIVE-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255873#comment-15255873 ] Hive QA commented on HIVE-12837: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800311/HIVE-12837.4.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 51 failed/errored test(s), 9869 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniLlapCliDriver - did not produce a TEST-*.xml file TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hadoop.hive.metastore.TestHiveMetaStoreWithEnvironmentContext.testEnvironmentContext org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec org.apache.hive.hcatalog.api.TestHCatClient.testDropTableException org.apache.hive.hcatalog.api.TestHCatClient.testEmptyTableInstantiation org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec org.apache.hive.hcatalog.api.TestHCatClient.testObjectNotFoundException org.apache.hive.hcatalog.api.TestHCatClient.testRenameTable org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure org.apache.hive.hcatalog.listener.TestDbNotificationListener.drop
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13421: --- Status: Patch Available (was: Open) Resubmitting for jenkins to pick > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13421: --- Fix Version/s: 2.1.0 Status: Open (was: Patch Available) > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13525) HoS hangs when job is empty
[ https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13525: -- Attachment: HIVE-13525.2.patch Upload same patch to run tests. > HoS hangs when job is empty > --- > > Key: HIVE-13525 > URL: https://issues.apache.org/jira/browse/HIVE-13525 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-13525.1.patch, HIVE-13525.2.patch, > HIVE-13525.2.patch > > > Observed in local tests. This should be the cause of HIVE-13402. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10293) enabling travis-ci build?
[ https://issues.apache.org/jira/browse/HIVE-10293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255791#comment-15255791 ] Gabor Liptak commented on HIVE-10293: - I'm still stumped with this ... I added a new repository with the correct Maven URL, but it is not getting used. https://repository.apache.org/service/local/repositories/releases/content/ https://travis-ci.org/gliptak/hive/builds/125466372 [INFO] [INFO] [INFO] Building Hive Service 2.1.0-SNAPSHOT [INFO] Downloading: https://oss.sonatype.org/content/repositories/releases/org/apache/directory/client/ldap/ldap-client-api/1.0.0-M14/ldap-client-api-1.0.0-M14.pom Downloading: https://repository.apache.org/releases/org/apache/directory/client/ldap/ldap-client-api/1.0.0-M14/ldap-client-api-1.0.0-M14.pom [WARNING] The POM for org.apache.directory.client.ldap:ldap-client-api:jar:1.0.0-M14 is missing, no dependency information available Downloading: https://nexus.codehaus.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/maven-metadata.xml Downloading: https://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/maven-metadata.xml [WARNING] Could not transfer metadata org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from/to codehaus-snapshots (https://nexus.codehaus.org/snapshots/): nexus.codehaus.org [WARNING] Failure to transfer org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from https://nexus.codehaus.org/snapshots/ was cached in the local repository, resolution will not be reattempted until the update interval of codehaus-snapshots has elapsed or updates are forced. Original error: Could not transfer metadata org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from/to codehaus-snapshots (https://nexus.codehaus.org/snapshots/): nexus.codehaus.org Downloading: https://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/ldap-client-api-0.1-SNAPSHOT.pom Downloading: https://nexus.codehaus.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/ldap-client-api-0.1-SNAPSHOT.pom > enabling travis-ci build? > - > > Key: HIVE-10293 > URL: https://issues.apache.org/jira/browse/HIVE-10293 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Gabor Liptak >Assignee: Gabor Liptak >Priority: Minor > Attachments: HIVE-10293.1.patch, HIVE-10293.2.diff > > > I would like to contribute a .travis.yml for Hive. > In particular, this would allow contributors working through Github, to > validate their own commits on their own branches. > Please comment. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13239) "java.lang.OutOfMemoryError: unable to create new native thread" occurs at Hive on Tez
[ https://issues.apache.org/jira/browse/HIVE-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wataru Yukawa resolved HIVE-13239. -- Resolution: Fixed > "java.lang.OutOfMemoryError: unable to create new native thread" occurs at > Hive on Tez > -- > > Key: HIVE-13239 > URL: https://issues.apache.org/jira/browse/HIVE-13239 > Project: Hive > Issue Type: Bug > Components: HiveServer2 > Environment: HDP2.3.4 > JDK1.8 > CentOS 6 >Reporter: Wataru Yukawa > > "ps -L $(pgrep -f hiveserver2) | wc -l" is more than 15,000 > HiveServer2 memory leak occurs. > hive query > {code} > FROM hoge_tmp > INSERT INTO TABLE hoge PARTITION (...) >SELECT ... WHERE ... > {code} > stacktrace > {code} > org.apache.hive.service.cli.HiveSQLException: Error while processing > statement: FAILED: Execution Error, return code -101 from > org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread > at > org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156) > at > org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:257) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:410) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:391) > at > org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:714) > at > org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:2238) > at > org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1753) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638) > at > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) > at > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:577) > at > org.apache.tez.common.TezCommonUtils.createFileForAM(TezCommonUtils.java:310) > at > org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:559) > at org.apache.tez.client.TezClient.start(TezClient.java:395) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java
[jira] [Commented] (HIVE-13239) "java.lang.OutOfMemoryError: unable to create new native thread" occurs at Hive on Tez
[ https://issues.apache.org/jira/browse/HIVE-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255788#comment-15255788 ] Wataru Yukawa commented on HIVE-13239: -- After upgrading to HDP2.4, this problem doesn't occur. So, I close this issue. Thanks. > "java.lang.OutOfMemoryError: unable to create new native thread" occurs at > Hive on Tez > -- > > Key: HIVE-13239 > URL: https://issues.apache.org/jira/browse/HIVE-13239 > Project: Hive > Issue Type: Bug > Components: HiveServer2 > Environment: HDP2.3.4 > JDK1.8 > CentOS 6 >Reporter: Wataru Yukawa > > "ps -L $(pgrep -f hiveserver2) | wc -l" is more than 15,000 > HiveServer2 memory leak occurs. > hive query > {code} > FROM hoge_tmp > INSERT INTO TABLE hoge PARTITION (...) >SELECT ... WHERE ... > {code} > stacktrace > {code} > org.apache.hive.service.cli.HiveSQLException: Error while processing > statement: FAILED: Execution Error, return code -101 from > org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread > at > org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156) > at > org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:257) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:410) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:391) > at > org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:714) > at > org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:2238) > at > org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1753) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638) > at > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) > at > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:577) > at > org.apache.tez.common.TezCommonUtils.createFileForAM(TezCommonUtils.java:310) > at > org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:559) > at org.apache.tez.client.TezClient.start(TezClient.java:395) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hiv
[jira] [Commented] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[ https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255761#comment-15255761 ] Hive QA commented on HIVE-7693: --- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12771809/HIVE-7693.07.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/67/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/67/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-67/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-67/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive a42bc67..413645d master -> origin/master + git reset --hard HEAD HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth) + git clean -f -d Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig + git checkout master Already on 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. + git reset --hard origin/master HEAD is now at 413645d HIVE-13570: Some queries with Union all fail when CBO is off (Yongzhi Chen, reviewed by Chaoyu Tang) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12771809 - PreCommit-HIVE-MASTER-Build > Invalid column ref error in order by when using column alias in select clause > and using having > -- > > Key: HIVE-7693 > URL: https://issues.apache.org/jira/browse/HIVE-7693 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0 >Reporter: Deepesh Khandelwal >Assignee: Pengcheng Xiong > Attachments: HIVE-7693.01.patch, HIVE-7693.02.patch, > HIVE-7693.03.patch, HIVE-7693.04.patch, HIVE-7693.05.patch, > HIVE-7693.06.patch, HIVE-7693.07.patch > > > Hive CLI session: > {noformat} > hive> create table abc(foo int, bar string); > OK > Time taken: 0.633 seconds > hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar > like '%abc%' order by foo; > FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or > column reference 'foo': (possible column names are: c0, c1) > {noformat} > Without having clause, the query runs fine, example: > {code} > select foo as c0, count(*) as c1 from abc group by foo, bar order by foo; > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13346) LLAP doesn't update metadata priority when reusing from cache; some tweaks in LRFU policy
[ https://issues.apache.org/jira/browse/HIVE-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255759#comment-15255759 ] Hive QA commented on HIVE-13346: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800298/HIVE-13346.02.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 57 failed/errored test(s), 9943 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5 org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityPreemption org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.lockConflictDbTable org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.
[jira] [Commented] (HIVE-10293) enabling travis-ci build?
[ https://issues.apache.org/jira/browse/HIVE-10293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255756#comment-15255756 ] Gabor Liptak commented on HIVE-10293: - https://docs.travis-ci.com/user/ci-environment/ > enabling travis-ci build? > - > > Key: HIVE-10293 > URL: https://issues.apache.org/jira/browse/HIVE-10293 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Gabor Liptak >Assignee: Gabor Liptak >Priority: Minor > Attachments: HIVE-10293.1.patch, HIVE-10293.2.diff > > > I would like to contribute a .travis.yml for Hive. > In particular, this would allow contributors working through Github, to > validate their own commits on their own branches. > Please comment. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10293) enabling travis-ci build?
[ https://issues.apache.org/jira/browse/HIVE-10293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255754#comment-15255754 ] Gabor Liptak commented on HIVE-10293: - https://github.com/travis-ci/travis-ci/issues/2677 > enabling travis-ci build? > - > > Key: HIVE-10293 > URL: https://issues.apache.org/jira/browse/HIVE-10293 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Gabor Liptak >Assignee: Gabor Liptak >Priority: Minor > Attachments: HIVE-10293.1.patch, HIVE-10293.2.diff > > > I would like to contribute a .travis.yml for Hive. > In particular, this would allow contributors working through Github, to > validate their own commits on their own branches. > Please comment. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13570) Some queries with Union all fail when CBO is off
[ https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-13570: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Confirmed the Llap failures are not related for precommit build #55 for HIVE-13494 has the exact same failures. Committed to master branch. > Some queries with Union all fail when CBO is off > > > Key: HIVE-13570 > URL: https://issues.apache.org/jira/browse/HIVE-13570 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.1.0 > > Attachments: HIVE-13570.1.PATCH, HIVE-13570.2.patch > > > Some queries with union all throws IndexOutOfBoundsException > when: > set hive.cbo.enable=false; > set hive.ppd.remove.duplicatefilters=true; > The stack is as: > {noformat} > java.lang.IndexOutOfBoundsException: Index: 67, Size: 67 > at java.util.ArrayList.rangeCheck(ArrayList.java:635) > at java.util.ArrayList.get(ArrayList.java:411) > at > org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.genColLists(ColumnPrunerProcCtx.java:161) > > at > org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.handleFilterUnionChildren(ColumnPrunerProcCtx.java:273) > > at > org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerFilterProc.process(ColumnPrunerProcFactory.java:108) > > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) > > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) > > at > org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172) > > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) > > at > org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135) > > at > org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:198) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10327) > > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192) > > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222) > > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:432) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1119) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1167) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1055) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207) > at > org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305) > at > org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403) > at > org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine, can break HiveQA
[ https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255731#comment-15255731 ] Hive QA commented on HIVE-11887: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800281/HIVE-11887.02.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 9952 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/65/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/65/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-65/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 38 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12800281 - PreCommit-HIVE-MASTER-Build > spark tests break
[jira] [Commented] (HIVE-13391) add an option to LLAP to use keytab to authenticate to read data
[ https://issues.apache.org/jira/browse/HIVE-13391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255710#comment-15255710 ] Hive QA commented on HIVE-13391: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800278/HIVE-13391.05.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/64/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/64/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-64/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-64/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth) + git clean -f -d Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig + git checkout master Already on 'master' + git reset --hard origin/master HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12800278 - PreCommit-HIVE-MASTER-Build > add an option to LLAP to use keytab to authenticate to read data > > > Key: HIVE-13391 > URL: https://issues.apache.org/jira/browse/HIVE-13391 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13391.01.patch, HIVE-13391.02.patch, > HIVE-13391.03.patch, HIVE-13391.04.patch, HIVE-13391.05.patch, > HIVE-13391.patch > > > This can be used for non-doAs case to allow access to clients who don't > propagate HDFS tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255709#comment-15255709 ] Hive QA commented on HIVE-13449: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800357/HIVE-13449.02.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 9900 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_varchar_4.q-smb_cache.q-tez_join_hash.q-and-8-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vectorized_parquet.q-vector_decimal_aggregate.q-tez_self_join.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityPreemption org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testSimpleTable org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler org.apache.hive.jdbc.TestSSL.testSSLFetchHttp org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/63/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/63/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-63/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests ex
[jira] [Updated] (HIVE-13601) Deadline is not registered at some places in HiveMetastore before the ObjectStore/MetaStoreDirectSql APIs are invoked
[ https://issues.apache.org/jira/browse/HIVE-13601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13601: - Summary: Deadline is not registered at some places in HiveMetastore before the ObjectStore/MetaStoreDirectSql APIs are invoked (was: Deadline is not registered at some places in HiveMetastore before the ObjectStore/MetaStoreDirectSQL APIs are invoked) > Deadline is not registered at some places in HiveMetastore before the > ObjectStore/MetaStoreDirectSql APIs are invoked > - > > Key: HIVE-13601 > URL: https://issues.apache.org/jira/browse/HIVE-13601 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > > HIVE-13542 exposed this issue as shown below when making Metastore API calls > via CBO code path. > {code} > 16-04-21T11:45:55,201 DEBUG [f2995fd1-e4fc-4783-86cb-9264f715e540 main[]]: > metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:timingTrace(864)) - > Direct SQL query in 14.825303ms + 0.086336ms, the query is [select > "COLUMN_NAME", "COLUMN_TYPE", "LONG_LOW_VALUE", "LONG_HIGH_VALUE", > "DOUBLE_LOW_VALU\ > E", "DOUBLE_HIGH_VALUE", "BIG_DECIMAL_LOW_VALUE", "BIG_DECIMAL_HIGH_VALUE", > "NUM_NULLS", "NUM_DISTINCTS", "AVG_COL_LEN", "MAX_COL_LEN", "NUM_TRUES", > "NUM_FALSES", "LAST_ANALYZED" from "TAB_COL_STATS" where "DB_NAME" = ? and > "TABLE_NAME" = ? and "COLUMN_NAME" in (...)] > 2016-04-21T11:45:55,242 WARN [f2995fd1-e4fc-4783-86cb-9264f715e540 main[]]: > metastore.ObjectStore (ObjectStore.java:handleDirectSqlError(2602)) - Direct > SQL failed, falling back to ORM > org.apache.hadoop.hive.metastore.api.MetaException: The threadlocal Deadline > is null, please register it first. > at > org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:178) > ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:149) > ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.makeColumnStats(MetaStoreDirectSql.java:1717) > ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:1201) > ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:6849) > ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:6846) > ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2567) > [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6845) > [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6839) > [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:4361) > [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:1613) > [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTableColumnStatistics(SessionHiveMetaStoreClient.java:347) > [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.metadata.Hive.getTableColumnStatistics(Hive.java:3317) > [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:758) > [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:302) > [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:417) > [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:410) > [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:158) > [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT] > at > org.apache.hadoop.hi
[jira] [Commented] (HIVE-13239) "java.lang.OutOfMemoryError: unable to create new native thread" occurs at Hive on Tez
[ https://issues.apache.org/jira/browse/HIVE-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255665#comment-15255665 ] Takuma Wakamori commented on HIVE-13239: Does this issue already resolved? ([~wyukawa] mentioned at HIVE-13273 that the OOM Error does not occur after upgrading to HDP2.4.) If not, I'll address this issue. I'm trying to fix bugs around hiveserver2. Thanks. > "java.lang.OutOfMemoryError: unable to create new native thread" occurs at > Hive on Tez > -- > > Key: HIVE-13239 > URL: https://issues.apache.org/jira/browse/HIVE-13239 > Project: Hive > Issue Type: Bug > Components: HiveServer2 > Environment: HDP2.3.4 > JDK1.8 > CentOS 6 >Reporter: Wataru Yukawa > > "ps -L $(pgrep -f hiveserver2) | wc -l" is more than 15,000 > HiveServer2 memory leak occurs. > hive query > {code} > FROM hoge_tmp > INSERT INTO TABLE hoge PARTITION (...) >SELECT ... WHERE ... > {code} > stacktrace > {code} > org.apache.hive.service.cli.HiveSQLException: Error while processing > statement: FAILED: Execution Error, return code -101 from > org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread > at > org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156) > at > org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:257) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:410) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:391) > at > org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) > at > org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:714) > at > org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:2238) > at > org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1753) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638) > at > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) > at > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:577) > at > org.apache.tez.common.TezCommonUtils.createFileForAM(TezCommonUtils.java:310) > at > org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:559) > at org.apache.tez.client.TezClient.start(TezClient.java:395) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.execu
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Attachment: HIVE-13341.07.patch > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch, HIVE-13341.07.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[ https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255649#comment-15255649 ] Hive QA commented on HIVE-9660: --- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800288/HIVE-9660.10.patch {color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 73 failed/errored test(s), 9947 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_file_dump org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_lengths org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge12 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge10 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge11 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge12 org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.ql.TestTxnCommands2.testBucketizedInputFormat org.apache.hadoop.hive.ql.TestTxnCommands2.testDeleteIn org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcNoPPD org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcPPD org.apache.hadoop.hive.ql.TestTxnCommands2.testUpdateMixedCase org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull org.apache.hadoop.hive.ql.io.orc.TestFileDump.testBloomFilter org.apache.hadoop.hive.ql.io.orc.TestFileDump.testBloomFilter2 org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDictionaryThreshold org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDump org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.lockConflictDbTable org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.q
[jira] [Commented] (HIVE-13592) metastore calls map is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-13592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255608#comment-15255608 ] Hive QA commented on HIVE-13592: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800271/HIVE-13592.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/61/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/61/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-61/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-61/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth) + git clean -f -d Removing pom.xml.orig + git checkout master Already on 'master' + git reset --hard origin/master HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12800271 - PreCommit-HIVE-MASTER-Build > metastore calls map is not thread safe > -- > > Key: HIVE-13592 > URL: https://issues.apache.org/jira/browse/HIVE-13592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13592.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255607#comment-15255607 ] Hive QA commented on HIVE-13097: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800265/HIVE-13097.03.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 9944 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/60/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/60/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-60/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 34 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12800265 - PreCommit-HIVE-MASTER-Build > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.01.patch, HIVE-13097.02.patch, > HIVE-13097.03.patch, HIVE-13097.versonChange.patch > >
[jira] [Commented] (HIVE-13586) Allow UDFs/UDTFs to indicate whether they can run with fetch task conversion
[ https://issues.apache.org/jira/browse/HIVE-13586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255573#comment-15255573 ] Hive QA commented on HIVE-13586: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800261/HIVE-13586.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 45 failed/errored test(s), 9922 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-cte_4.q-schema_evol_text_nonvec_mapwork_table.q-vector_groupby_reduce.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_coalesce.q-auto_sortmerge_join_7.q-orc_merge9.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alias_casted_column org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_simple_select org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constant_prop org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_allchildsarenull org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_coalesce org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_named_struct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_struct org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs org.apache.hive.hcatalog.listener.TestDbNotificationListener.sqlInsertPartition org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/59/testReport Console output: http://ec2-54-177
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: (was: HIVE-12878.09.patch) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: HIVE-12878.09.patch > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13258) LLAP: Add hdfs bytes read and spilled bytes to tez print summary
[ https://issues.apache.org/jira/browse/HIVE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255541#comment-15255541 ] Hive QA commented on HIVE-13258: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800260/HIVE-13258.1.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/58/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/58/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-58/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-58/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive 7419768..a42bc67 master -> origin/master + git reset --hard HEAD HEAD is now at 7419768 HIVE-13527: Using deprecated APIs in HBase client causes zookeeper connection leaks (Naveen Gangam via Chaoyu Tang) + git clean -f -d Removing ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleConstantReduction.java Removing ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregatePullUpConstantsRule.java + git checkout master Already on 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. + git reset --hard origin/master HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12800260 - PreCommit-HIVE-MASTER-Build > LLAP: Add hdfs bytes read and spilled bytes to tez print summary > > > Key: HIVE-13258 > URL: https://issues.apache.org/jira/browse/HIVE-13258 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13258.1.patch, HIVE-13258.1.patch, > llap-fs-counters-full-cache-hit.png, llap-fs-counters.png > > > When printing counters to console it will be useful to print hdfs bytes read > and spilled bytes which will help with debugging issues faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255540#comment-15255540 ] Hive QA commented on HIVE-13068: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800258/HIVE-13068.01.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 263 failed/errored test(s), 9930 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_grouping_sets.q-mapjoin_mapjoin.q-update_all_partitioned.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguitycheck org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ansi_sql_arithmetic org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_groupby org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast_on_constant org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_SortUnionTransposeRule org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_const org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_pad_convert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropWhen org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog_semijoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_sel org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_colname org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_mat_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_mat_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_rdd_cache org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_cond_pushdown org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_eq_with_case_when org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_when org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_folder_predicate org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_foldts org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_duplicate_key org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppd org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_unused org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_interval_arithmetic org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDri
[jira] [Updated] (HIVE-13176) OutOfMemoryError : GC overhead limit exceeded
[ https://issues.apache.org/jira/browse/HIVE-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuma Wakamori updated HIVE-13176: --- Description: KDetected leaks while testing hiveserver2 concurrency setup with LLAP. 2016-02-26T12:50:58,131 ERROR [HiveServer2-Background-Pool: Thread-311030]: operation.Operation (SQLOperation.java:run(230)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.StatsTask. GC overhead limit exceeded at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:333) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:177) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:227) [hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_45] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_45] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.1.2.3.5.1-36.jar:?] at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:239) [hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45] was: Detected leaks while testing hiveserver2 concurrency setup with LLAP. 2016-02-26T12:50:58,131 ERROR [HiveServer2-Background-Pool: Thread-311030]: operation.Operation (SQLOperation.java:run(230)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.StatsTask. GC overhead limit exceeded at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:333) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:177) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:227) [hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_45] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_45] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.1.2.3.5.1-36.jar:?] at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:239) [hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45] > OutOfMemoryError : GC overhead limit exceeded > -- > > Key: HIVE-13176 > URL: https://issues.apache.org/jira/browse/HIVE-13176 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Kavan Suresh >Assignee: Siddharth Seth > Attachments: dataNucleus.png, fs.png, shutdownhook.png > > > KDetected leaks while testing hiveserver2 concurrency setup with LLAP. > 2016-02-26T12:50:58,131 ERROR [HiveServer2-Background-Pool: Thread-311030]: > operation.Operation (SQLOperation.java:run(230)) - Error running hive query: > org.apache.hive.service.cli.HiveSQLException: Error while processing > statement: FAILED: Execution Error, return code -101 from > org.apache.hadoop.hive.ql.exec.StatsTask. GC over
[jira] [Commented] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255533#comment-15255533 ] Lefty Leverenz commented on HIVE-13494: --- Doc note: This removes *hive.llap.queue.metrics.percentiles.intervals* and adds *hive.llap.io.decoding.metrics.percentiles.intervals* so the changes need to be documented in the LLAP section of Configuration Properties for release 2.1.0. * [Configuration Properties -- LLAP | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP] Question: Why is *hive.llap.queue.metrics.percentiles.intervals* included at the end of the description for *hive.llap.io.decoding.metrics.percentiles.intervals* (or are "\n" and "+" on the previous line errors)? {code} - LLAP_QUEUE_METRICS_PERCENTILE_INTERVALS("hive.llap.queue.metrics.percentiles.intervals", "", + LLAP_IO_DECODING_METRICS_PERCENTILE_INTERVALS("hive.llap.io.decoding.metrics.percentiles.intervals", "30", "Comma-delimited set of integers denoting the desired rollover intervals (in seconds)\n" + -"for percentile latency metrics on the LLAP daemon producer-consumer queue.\n" + -"By default, percentile latency metrics are disabled."), +"for percentile latency metrics on the LLAP daemon IO decoding time.\n" + +"hive.llap.queue.metrics.percentiles.intervals"), {code} > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13494: -- Labels: TODOC2.1 (was: ) > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13536: - Attachment: HIVE-13536.2.patch Addressed [~sseth]'s review comments. > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13536: - Status: Patch Available (was: Open) > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13494: - Resolution: Fixed Fix Version/s: 2.1.0 Target Version/s: 2.1.0 (was: 2.1.0, 2.0.1) Status: Resolved (was: Patch Available) Committed to master. Thanks [~sseth] for the review! > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.1.0 > > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255524#comment-15255524 ] Prasanth Jayachandran commented on HIVE-13494: -- These test failure does not look related. I ran all the llap test on local machine and all of them passed. It's probably some intermittent issue that caused these test to fails on precommit. > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13480) Add hadoop2 metrics reporter for Codahale metrics
[ https://issues.apache.org/jira/browse/HIVE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255520#comment-15255520 ] Lefty Leverenz commented on HIVE-13480: --- Doc note: This adds *hive.service.metrics.hadoop2.frequency* and *hive.service.metrics.hadoop2.component* to HiveConf.java and adds HADOOP2 to the description of *hive.service.metrics.reporter*. They need to be documented in the Metrics section of Configuration Properties for release 2.1.0. * [Configuration Properties -- Metrics | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Metrics] > Add hadoop2 metrics reporter for Codahale metrics > - > > Key: HIVE-13480 > URL: https://issues.apache.org/jira/browse/HIVE-13480 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13480.2.patch, HIVE-13480.3.patch, > HIVE-13480.4.patch, HIVE-13480.5.patch, HIVE-13480.patch > > > Multiple other apache components allow sending metrics over to Hadoop2 > metrics, which allow for monitoring solutions like Ambari Metrics Server to > work against that to show metrics for components in one place. Our Codahale > metrics works very well, so ideally, we would like to bridge the two, to > allow Codahale to add a Hadoop2 reporter that enables us to continue to use > Codahale metrics (i.e. not write another custom metrics impl) but report > using Hadoop2. > Apache Phoenix also had such a recent usecase and were in the process of > adding in a stub piece that allows this forwarding. We should use the same > reporter to minimize redundancy while pushing metrics to a centralized > solution like Hadoop2 Metrics/AMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13494: - Attachment: (was: HIVE-13494.2.patch) > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13480) Add hadoop2 metrics reporter for Codahale metrics
[ https://issues.apache.org/jira/browse/HIVE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13480: -- Labels: TODOC2.1 (was: ) > Add hadoop2 metrics reporter for Codahale metrics > - > > Key: HIVE-13480 > URL: https://issues.apache.org/jira/browse/HIVE-13480 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13480.2.patch, HIVE-13480.3.patch, > HIVE-13480.4.patch, HIVE-13480.5.patch, HIVE-13480.patch > > > Multiple other apache components allow sending metrics over to Hadoop2 > metrics, which allow for monitoring solutions like Ambari Metrics Server to > work against that to show metrics for components in one place. Our Codahale > metrics works very well, so ideally, we would like to bridge the two, to > allow Codahale to add a Hadoop2 reporter that enables us to continue to use > Codahale metrics (i.e. not write another custom metrics impl) but report > using Hadoop2. > Apache Phoenix also had such a recent usecase and were in the process of > adding in a stub piece that allows this forwarding. We should use the same > reporter to minimize redundancy while pushing metrics to a centralized > solution like Hadoop2 Metrics/AMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)