[jira] [Updated] (HIVE-11526) LLAP: implement LLAP UI as a separate service
[ https://issues.apache.org/jira/browse/HIVE-11526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuya OZAWA updated HIVE-11526: -- Status: Patch Available (was: Open) > LLAP: implement LLAP UI as a separate service > - > > Key: HIVE-11526 > URL: https://issues.apache.org/jira/browse/HIVE-11526 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Sergey Shelukhin >Assignee: Yuya OZAWA > Attachments: HIVE-11526.2.patch, HIVE-11526.3.patch, > HIVE-11526.patch, llap_monitor_design.pdf > > > The specifics are vague at this point. > Hadoop metrics can be output, as well as metrics we collect and output in > jmx, as well as those we collect per fragment and log right now. > This service can do LLAP-specific views, and per-query aggregation. > [~gopalv] may have some information on how to reuse existing solutions for > part of the work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12967) Change LlapServiceDriver to read a properties file instead of llap-daemon-site
[ https://issues.apache.org/jira/browse/HIVE-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1515#comment-1515 ] Lefty Leverenz commented on HIVE-12967: --- Doc note: This adds a description to *hive.llap.daemon.yarn.container.mb* in HiveConf.java. When a new section for LLAP configuration parameters is added to the wiki, the new description needs to be included. * [Configuration Properties | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties] Should this also be discussed in the LLAP documentation? Currently that's just a design doc attached to HIVE-7926. * [Live long and process (#LLAP) | https://issues.apache.org/jira/secure/attachment/12665704/LLAPdesigndocument.pdf] > Change LlapServiceDriver to read a properties file instead of llap-daemon-site > -- > > Key: HIVE-12967 > URL: https://issues.apache.org/jira/browse/HIVE-12967 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-12967.01.patch, HIVE-12967.1.wip.txt, > HIVE-12967.2.patch > > > Having a copy of llap-daemon-site on the client node can be quite confusing, > since LlapServiceDriver generates the actual llap-daemon-site used by daemons. > Instead of this - base settings can be picked up from a properties file. > Also add java_home as a parameter to the script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11526) LLAP: implement LLAP UI as a separate service
[ https://issues.apache.org/jira/browse/HIVE-11526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-11526: --- Attachment: HIVE-11526.3.patch [~yozawa]: The .2 patch looks good and works on my cluster - LGTM +1. For the purpose of reducing the size of the patch, I have converted the common files between service/ and llap-server/ into symlinks. And to maintain licensing reqs, I have included a LICENSE entry for jquery sparklines. Have renamed metrics.html -> index.html in prep for HIVE-12942 Thank you for the contribution and this will be committed into master, once tests run. > LLAP: implement LLAP UI as a separate service > - > > Key: HIVE-11526 > URL: https://issues.apache.org/jira/browse/HIVE-11526 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Sergey Shelukhin >Assignee: Yuya OZAWA > Attachments: HIVE-11526.2.patch, HIVE-11526.3.patch, > HIVE-11526.patch, llap_monitor_design.pdf > > > The specifics are vague at this point. > Hadoop metrics can be output, as well as metrics we collect and output in > jmx, as well as those we collect per fragment and log right now. > This service can do LLAP-specific views, and per-query aggregation. > [~gopalv] may have some information on how to reuse existing solutions for > part of the work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13015) Bundle Log4j2 jars with hive-exec
[
https://issues.apache.org/jira/browse/HIVE-13015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149985#comment-15149985
]
Hive QA commented on HIVE-13015:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788085/HIVE-13015.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 590 failed/errored test(s), 9790 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-orc_vectorization_ppd.q-vector_left_outer_join2.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_add_part_multiple
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join24
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join28
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join31
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_5
[jira] [Updated] (HIVE-12967) Change LlapServiceDriver to read a properties file instead of llap-daemon-site
[ https://issues.apache.org/jira/browse/HIVE-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12967: -- Labels: TODOC2.1 (was: ) > Change LlapServiceDriver to read a properties file instead of llap-daemon-site > -- > > Key: HIVE-12967 > URL: https://issues.apache.org/jira/browse/HIVE-12967 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-12967.01.patch, HIVE-12967.1.wip.txt, > HIVE-12967.2.patch > > > Having a copy of llap-daemon-site on the client node can be quite confusing, > since LlapServiceDriver generates the actual llap-daemon-site used by daemons. > Instead of this - base settings can be picked up from a properties file. > Also add java_home as a parameter to the script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12967) Change LlapServiceDriver to read a properties file instead of llap-daemon-site
[ https://issues.apache.org/jira/browse/HIVE-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-12967: --- Resolution: Fixed Release Note: HIVE-12967: Change LlapServiceDriver to read a properties file instead of llap-daemon-site (Siddarth Seth, via Gopal V) Status: Resolved (was: Patch Available) > Change LlapServiceDriver to read a properties file instead of llap-daemon-site > -- > > Key: HIVE-12967 > URL: https://issues.apache.org/jira/browse/HIVE-12967 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-12967.01.patch, HIVE-12967.1.wip.txt, > HIVE-12967.2.patch > > > Having a copy of llap-daemon-site on the client node can be quite confusing, > since LlapServiceDriver generates the actual llap-daemon-site used by daemons. > Instead of this - base settings can be picked up from a properties file. > Also add java_home as a parameter to the script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12967) Change LlapServiceDriver to read a properties file instead of llap-daemon-site
[ https://issues.apache.org/jira/browse/HIVE-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-12967: --- Fix Version/s: 2.1.0 > Change LlapServiceDriver to read a properties file instead of llap-daemon-site > -- > > Key: HIVE-12967 > URL: https://issues.apache.org/jira/browse/HIVE-12967 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-12967.01.patch, HIVE-12967.1.wip.txt, > HIVE-12967.2.patch > > > Having a copy of llap-daemon-site on the client node can be quite confusing, > since LlapServiceDriver generates the actual llap-daemon-site used by daemons. > Instead of this - base settings can be picked up from a properties file. > Also add java_home as a parameter to the script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13056) delegation tokens do not work with HS2 when used with http transport and kerberos
[
https://issues.apache.org/jira/browse/HIVE-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149791#comment-15149791
]
Hive QA commented on HIVE-13056:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12787794/HIVE-13056.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9790 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7005/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7005/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7005/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12787794 - PreCommit-HIVE-TRUNK-Build
> delegation tokens do not work with HS2 when used with http transport and
> kerberos
> -
>
> Key: HIVE-13056
> URL: https://issues.apache.org/jira/browse/HIVE-13056
> Project: Hive
> Issue Type: Bug
> Components: Authentication
>Affects Versions: 1.2.1
>Reporter: Cheng Xu
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Attachments: HIVE-13056.patch
>
>
> We're getting a HiveSQLException on secure windows clusters.
> {code}
> 2016-02-08
> 13:48:09,535|beaver.machine|INFO|6114|140264674350912|MainThread|Job ID :
> 000-160208134528402-oozie-oozi-W
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|Workflow
> Name : hive2-wf
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|App Path
> :
> wasb://oozie1-hb...@humbtestings5jp.blob.core.windows.net/user/hrt_qa/test_hiveserver2
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|Status
> : KILLED
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|Run
> : 0
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|User
> : hrt_qa
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|Group
> : -
> 2016-02-08
> 13:48:09,547|beaver.machine|INFO|6114|140264674350912|MainThread|Created
> : 2016-02-08 13:47 GMT
> 2016-02-08
> 13:48:09,548|beaver.machine|INFO|6114|140264674350912|MainThread|Started
> : 2016-02-08 13:47 GMT
> 2016-02-08
> 13:48:09,552|beaver.machine|INFO|6114|140264674350912|MainThread|Last
> Modified : 2016-02-08 13:48 GMT
> 2016-02-08
> 13:48:09,553|beaver.machine|INFO|6114|140264674350912|MainThread|Ended
> : 2016-02-08 13:48 GMT
> 2016-02-08
> 13:48:09,553|beaver.machine|INFO|6114|140264674350912|MainThread|CoordAction
> ID: -
> 2016-02-08 13:48:09,566|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,566|beaver.machine|INFO|6114|140264674350912|MainThread|Actions
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|ID
> Status
> Ext ID Ext Status Err Code
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,571|beaver.machine|INFO|6114|140264674350912|MainThread|000-160208134528402-oozie-oozi-W@:start:
> OK- OK
[jira] [Updated] (HIVE-12927) HBase metastore: sequences should be one per row, not all in one row
[
https://issues.apache.org/jira/browse/HIVE-12927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-12927:
--
Attachment: HIVE-12927.2.patch
New patch that corrects issues with sequences to be one per row instead of one
per column. Also switched new CURRENT_VERSION table to be sequence based
instead of its own table.
> HBase metastore: sequences should be one per row, not all in one row
>
>
> Key: HIVE-12927
> URL: https://issues.apache.org/jira/browse/HIVE-12927
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Sergey Shelukhin
>Assignee: Alan Gates
>Priority: Critical
> Attachments: HIVE-12927.2.patch, HIVE-12927.patch
>
>
> {noformat}
> long getNextSequence(byte[] sequence) throws IOException {
> {noformat}
> Is not safe in presence of any concurrency. It should use HBase increment API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13040) Handle empty bucket creations more efficiently
[ https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13040: Attachment: HIVE-13040.2.patch > Handle empty bucket creations more efficiently > --- > > Key: HIVE-13040 > URL: https://issues.apache.org/jira/browse/HIVE-13040 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13040.2.patch, HIVE-13040.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12856) LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)
[ https://issues.apache.org/jira/browse/HIVE-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12856: Attachment: HIVE-12856.02.nogen.patch > LLAP: update (add/remove) the UDFs available in LLAP when they are changed > (refresh periodically) > - > > Key: HIVE-12856 > URL: https://issues.apache.org/jira/browse/HIVE-12856 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12856.01.nogen.patch, HIVE-12856.01.patch, > HIVE-12856.02.nogen.patch, HIVE-12856.02.patch, HIVE-12856.nogen.patch, > HIVE-12856.patch > > > I don't think re-querying the functions is going to scale, and the sessions > obviously cannot notify all LLAP clusters of every change. We should add > global versioning to metastore functions to track changes, and then possibly > add a notification mechanism, potentially thru ZK to avoid overloading the > metastore itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12856) LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)
[ https://issues.apache.org/jira/browse/HIVE-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12856: Attachment: HIVE-12856.02.patch > LLAP: update (add/remove) the UDFs available in LLAP when they are changed > (refresh periodically) > - > > Key: HIVE-12856 > URL: https://issues.apache.org/jira/browse/HIVE-12856 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12856.01.nogen.patch, HIVE-12856.01.patch, > HIVE-12856.02.nogen.patch, HIVE-12856.02.patch, HIVE-12856.nogen.patch, > HIVE-12856.patch > > > I don't think re-querying the functions is going to scale, and the sessions > obviously cannot notify all LLAP clusters of every change. We should add > global versioning to metastore functions to track changes, and then possibly > add a notification mechanism, potentially thru ZK to avoid overloading the > metastore itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13007) add an API to force reload UDFs to LLAP (either force reload everything, or force a regular refresh)
[ https://issues.apache.org/jira/browse/HIVE-13007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-13007: --- Assignee: Sergey Shelukhin > add an API to force reload UDFs to LLAP (either force reload everything, or > force a regular refresh) > > > Key: HIVE-13007 > URL: https://issues.apache.org/jira/browse/HIVE-13007 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table
[
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149606#comment-15149606
]
Sergey Shelukhin edited comment on HIVE-5795 at 2/17/16 12:50 AM:
--
This forces the entire input into a single split, which defeats the purpose of
using Hive in the first place. I would not recommend anyone to use this feature
except for experimentation. The headers/footers should be cleared as part of an
ETL process.
was (Author: sershe):
This forces the entire input into a single split, which defeats the purpose of
using Hive in the first place. I would not recommend anyone to use this feature
except for experimentation. The headers/footers should be cleared as part of
the ETL process.
> Hive should be able to skip header and footer rows when reading data file for
> a table
> -
>
> Key: HIVE-5795
> URL: https://issues.apache.org/jira/browse/HIVE-5795
> Project: Hive
> Issue Type: New Feature
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Labels: TODOC13
> Fix For: 0.13.0
>
> Attachments: HIVE-5795.1.patch, HIVE-5795.2.patch, HIVE-5795.3.patch,
> HIVE-5795.4.patch, HIVE-5795.5.patch
>
>
> Hive should be able to skip header and footer lines when reading data file
> from table. In this way, user don't need to processing data which generated
> by other application with a header or footer and directly use the file for
> table operations.
> To implement this, the idea is adding new properties in table descriptions to
> define the number of lines in header and footer and skip them when reading
> the record from record reader. An DDL example for creating a table with
> header and footer should be like this:
> {code}
> Create external table testtable (name string, message string) row format
> delimited fields terminated by '\t' lines terminated by '\n' location
> '/testtable' tblproperties ("skip.header.line.count"="1",
> "skip.footer.line.count"="2");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table
[
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149606#comment-15149606
]
Sergey Shelukhin edited comment on HIVE-5795 at 2/17/16 12:51 AM:
--
This forces the entire input into a single split, which defeats the purpose of
using Hive in the first place - might as well run the analysis on a local
machine. I would not recommend anyone to use this feature except for
experimentation. The headers/footers should be cleared as part of an ETL
process.
was (Author: sershe):
This forces the entire input into a single split, which defeats the purpose of
using Hive in the first place. I would not recommend anyone to use this feature
except for experimentation. The headers/footers should be cleared as part of an
ETL process.
> Hive should be able to skip header and footer rows when reading data file for
> a table
> -
>
> Key: HIVE-5795
> URL: https://issues.apache.org/jira/browse/HIVE-5795
> Project: Hive
> Issue Type: New Feature
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Labels: TODOC13
> Fix For: 0.13.0
>
> Attachments: HIVE-5795.1.patch, HIVE-5795.2.patch, HIVE-5795.3.patch,
> HIVE-5795.4.patch, HIVE-5795.5.patch
>
>
> Hive should be able to skip header and footer lines when reading data file
> from table. In this way, user don't need to processing data which generated
> by other application with a header or footer and directly use the file for
> table operations.
> To implement this, the idea is adding new properties in table descriptions to
> define the number of lines in header and footer and skip them when reading
> the record from record reader. An DDL example for creating a table with
> header and footer should be like this:
> {code}
> Create external table testtable (name string, message string) row format
> delimited fields terminated by '\t' lines terminated by '\n' location
> '/testtable' tblproperties ("skip.header.line.count"="1",
> "skip.footer.line.count"="2");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table
[
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149606#comment-15149606
]
Sergey Shelukhin commented on HIVE-5795:
This forces the entire input into a single split, which defeats the purpose of
using Hive in the first place. I would not recommend anyone to use this feature
except for experimentation. The headers/footers should be cleared as part of
the ETL process.
> Hive should be able to skip header and footer rows when reading data file for
> a table
> -
>
> Key: HIVE-5795
> URL: https://issues.apache.org/jira/browse/HIVE-5795
> Project: Hive
> Issue Type: New Feature
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Labels: TODOC13
> Fix For: 0.13.0
>
> Attachments: HIVE-5795.1.patch, HIVE-5795.2.patch, HIVE-5795.3.patch,
> HIVE-5795.4.patch, HIVE-5795.5.patch
>
>
> Hive should be able to skip header and footer lines when reading data file
> from table. In this way, user don't need to processing data which generated
> by other application with a header or footer and directly use the file for
> table operations.
> To implement this, the idea is adding new properties in table descriptions to
> define the number of lines in header and footer and skip them when reading
> the record from record reader. An DDL example for creating a table with
> header and footer should be like this:
> {code}
> Create external table testtable (name string, message string) row format
> delimited fields terminated by '\t' lines terminated by '\n' location
> '/testtable' tblproperties ("skip.header.line.count"="1",
> "skip.footer.line.count"="2");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149587#comment-15149587
]
Hive QA commented on HIVE-11160:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788058/HIVE-11160.04.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9794 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-unionDistinct_1.q-insert_update_delete.q-selectDistinctStar.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.hbase.TestHBaseAggregateStatsCacheWithBitVector.allPartitions
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7004/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7004/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7004/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788058 - PreCommit-HIVE-TRUNK-Build
> Auto-gather column stats
>
>
> Key: HIVE-11160
> URL: https://issues.apache.org/jira/browse/HIVE-11160
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch,
> HIVE-11160.03.patch, HIVE-11160.04.patch
>
>
> Hive will collect table stats when set hive.stats.autogather=true during the
> INSERT OVERWRITE command. And then the users need to collect the column stats
> themselves using "Analyze" command. In this patch, the column stats will also
> be collected automatically. More specifically, INSERT OVERWRITE will
> automatically create new column stats. INSERT INTO will automatically merge
> new column stats with existing ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12967) Change LlapServiceDriver to read a properties file instead of llap-daemon-site
[ https://issues.apache.org/jira/browse/HIVE-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149548#comment-15149548 ] Gopal V commented on HIVE-12967: Test failures are unrelated. > Change LlapServiceDriver to read a properties file instead of llap-daemon-site > -- > > Key: HIVE-12967 > URL: https://issues.apache.org/jira/browse/HIVE-12967 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12967.01.patch, HIVE-12967.1.wip.txt, > HIVE-12967.2.patch > > > Having a copy of llap-daemon-site on the client node can be quite confusing, > since LlapServiceDriver generates the actual llap-daemon-site used by daemons. > Instead of this - base settings can be picked up from a properties file. > Also add java_home as a parameter to the script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13057) Remove duplicate copies of TableDesc property values in PartitionDesc
[ https://issues.apache.org/jira/browse/HIVE-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149543#comment-15149543 ] Mohit Sabharwal commented on HIVE-13057: Test failures are unrelated. (Also occur in unrelated runs like: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6976/#showFailuresLink) > Remove duplicate copies of TableDesc property values in PartitionDesc > - > > Key: HIVE-13057 > URL: https://issues.apache.org/jira/browse/HIVE-13057 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13057.patch > > > For a partitioned table, each PartitionDesc has a copy of corresponding > TableDesc. > While TableDesc is mutable and hence cannot be interned, it's property values > can be. > For a simple select on a table with 100K partitions, this cut total number of > String instances by ~65%. > Most replicated strings were location, serde, input/output format, column, > types, table name, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-2409) Semicolons in strings/comments are parsed as query-ending.
[ https://issues.apache.org/jira/browse/HIVE-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149541#comment-15149541 ] Alan Jackoway commented on HIVE-2409: - This is still happening on hive and beeline, whether using a file as input or just typing in the command lines. > Semicolons in strings/comments are parsed as query-ending. > -- > > Key: HIVE-2409 > URL: https://issues.apache.org/jira/browse/HIVE-2409 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: Adam Kramer > > This fails: > select '.*? (f_.*?)[ ;$]' from akramer_one_row ; > This succeeds: > select '.*? (f_.*?)[ \;$]' from akramer_one_row ; > ...there is no reasonable syntactic structure that wuold require the escaping > of a semicolon in a '-marked string. The query parser should NOT split on > semicolons that are in strings OR in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13065) Hive throws NPE when writing map type data to a HBase backed table
[
https://issues.apache.org/jira/browse/HIVE-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149531#comment-15149531
]
Yibing Shi commented on HIVE-13065:
---
How about the reading part? If we skip the null values, would it affect the
reading part?
And what if we have a null value in key set? This is possible in theory.
> Hive throws NPE when writing map type data to a HBase backed table
> --
>
> Key: HIVE-13065
> URL: https://issues.apache.org/jira/browse/HIVE-13065
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13065.1.patch
>
>
> Hive throws NPE when writing data to a HBase backed table with below
> conditions:
> # There is a map type column
> # The map type column has NULL in its values
> Below are the reproduce steps:
> *1) Create a HBase backed Hive table*
> {code:sql}
> create table hbase_test (id bigint, data map)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with serdeproperties ("hbase.columns.mapping" = ":key,cf:map_col")
> tblproperties ("hbase.table.name" = "hive_test");
> {code}
> *2) insert data into above table*
> {code:sql}
> insert overwrite table hbase_test select 1 as id, map('abcd', null) as data
> from src limit 1;
> {code}
> The mapreduce job for insert query fails. Error messages are as below:
> {noformat}
> 2016-02-15 02:26:33,225 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0) {"key":{},"value":{"_col0":1,"_col1":{"abcd":null}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:265)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{},"value":{"_col0":1,"_col1":{"abcd":null}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:253)
> ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:731)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.LimitOperator.processOp(LimitOperator.java:51)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:244)
> ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:286)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:666)
> ... 14 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:221)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:275)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:282)
> ... 15 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13045) move guava dependency back to 14 after HIVE-12952
[ https://issues.apache.org/jira/browse/HIVE-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149499#comment-15149499 ] Mohit Sabharwal commented on HIVE-13045: Test failures are unrelated. (Most occur in unrelated run http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6976/#showFailuresLink. Remainder are infrastructural issues). Re-attaching patch for re-run to confirm. > move guava dependency back to 14 after HIVE-12952 > - > > Key: HIVE-13045 > URL: https://issues.apache.org/jira/browse/HIVE-13045 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13045.patch, HIVE-13045.patch > > > HIVE-12952 removed usage of EvictingQueue, so we don't need to up dependency > to guava 15 at this point - avoid version related conflicts with clients if > we can avoid it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13045) move guava dependency back to 14 after HIVE-12952
[ https://issues.apache.org/jira/browse/HIVE-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13045: --- Attachment: HIVE-13045.patch > move guava dependency back to 14 after HIVE-12952 > - > > Key: HIVE-13045 > URL: https://issues.apache.org/jira/browse/HIVE-13045 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13045.patch, HIVE-13045.patch > > > HIVE-12952 removed usage of EvictingQueue, so we don't need to up dependency > to guava 15 at this point - avoid version related conflicts with clients if > we can avoid it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149452#comment-15149452
]
Rohit Dholakia commented on HIVE-12049:
---
uploaded a new version of end to end patch. has some bug fixes and some changes
to the FileSinkOperator and ThriftJDBCSerDe.
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.2.patch,
> HIVE-12049.3.patch, HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohit Dholakia updated HIVE-12049:
--
Attachment: HIVE-12049.6.patch
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.2.patch,
> HIVE-12049.3.patch, HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12927) HBase metastore: sequences should be one per row, not all in one row
[
https://issues.apache.org/jira/browse/HIVE-12927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-12927:
--
Summary: HBase metastore: sequences should be one per row, not all in one
row (was: HBase metastore: sequences are not safe)
> HBase metastore: sequences should be one per row, not all in one row
>
>
> Key: HIVE-12927
> URL: https://issues.apache.org/jira/browse/HIVE-12927
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Sergey Shelukhin
>Assignee: Alan Gates
>Priority: Critical
> Attachments: HIVE-12927.patch
>
>
> {noformat}
> long getNextSequence(byte[] sequence) throws IOException {
> {noformat}
> Is not safe in presence of any concurrency. It should use HBase increment API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12064) prevent transactional=false
[ https://issues.apache.org/jira/browse/HIVE-12064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149437#comment-15149437 ] Alan Gates commented on HIVE-12064: --- In general looks good. A couple of minor issues: The error message ""The table must be bucketed and stored as ORC in order to be transactional" should be something like: "The table must be bucketed and stored using an ACID compliant format (such as ORC)". This way if a user implements their own format or other formats implement ACID the error message is still valid. When checking validity of a createTableStatement with false, you let the false through to avoid backwards compatibility issues. I agree this is the right choice, but there should be an LOG.info() there to tell users we're ignoring and dropping the value. > prevent transactional=false > --- > > Key: HIVE-12064 > URL: https://issues.apache.org/jira/browse/HIVE-12064 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-12064.2.patch, HIVE-12064.3.patch, > HIVE-12064.4.patch, HIVE-12064.patch > > > currently a tblproperty transactional=true must be set to make a table behave > in ACID compliant way. > This is misleading in that it seems like changing it to transactional=false > makes the table non-acid but on disk layout of acid table is different than > plain tables. So changing this property may cause wrong data to be returned. > Should prevent transactional=false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12612) beeline always exits with 0 status when reading query from standard input
[
https://issues.apache.org/jira/browse/HIVE-12612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149400#comment-15149400
]
Ruslan Dautkhanov commented on HIVE-12612:
--
Got it. Yep, that's correct. I see that beeline always returns 0 even for bad
input. Bummer. Thanks for submitting this.
> beeline always exits with 0 status when reading query from standard input
> -
>
> Key: HIVE-12612
> URL: https://issues.apache.org/jira/browse/HIVE-12612
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
> Environment: CDH5.5.0
>Reporter: Paulo Sequeira
>Priority: Minor
>
> Similar to what was reported on HIVE-6978, but now it only happens when the
> query is read from the standard input. For example, the following fails as
> expected:
> {code}
> bash$ if beeline -u "jdbc:hive2://..." -e "boo;" ; then echo "Ok?!" ; else
> echo "Failed!" ; fi
> Connecting to jdbc:hive2://...
> Connected to: Apache Hive (version 1.1.0-cdh5.5.0)
> Driver: Hive JDBC (version 1.1.0-cdh5.5.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:0
> cannot recognize input near 'boo' '' '' (state=42000,code=4)
> Closing: 0: jdbc:hive2://...
> Failed!
> {code}
> But the following does not:
> {code}
> bash$ if echo "boo;"|beeline -u "jdbc:hive2://..." ; then echo "Ok?!" ; else
> echo "Failed!" ; fi
> Connecting to jdbc:hive2://...
> Connected to: Apache Hive (version 1.1.0-cdh5.5.0)
> Driver: Hive JDBC (version 1.1.0-cdh5.5.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 1.1.0-cdh5.5.0 by Apache Hive
> 0: jdbc:hive2://...:8> Error: Error while compiling statement: FAILED:
> ParseException line 1:0 cannot recognize input near 'boo' '' ''
> (state=42000,code=4)
> 0: jdbc:hive2://...:8> Closing: 0: jdbc:hive2://...
> Ok?!
> {code}
> This was misleading our batch scripts to always believe that the execution of
> the queries succeded, when sometimes that was not the case.
> h2. Workaround
> We found we can work around the issue by always using the -e or the -f
> parameters, and even reading the standard input through the /dev/stdin device
> (this was useful because a lot of the scripts fed the queries from here
> documents), like this:
> {code:title=some-script.sh}
> #!/bin/sh
> set -o nounset -o errexit -o pipefail
> # As beeline is failing to report an error status if reading the query
> # to be executed from STDIN, check whether no -f or -e option is used
> # and, in that case, pretend it has to read the query from a regular
> # file using -f to read from /dev/stdin
> function beeline_workaround_exit_status () {
> for arg in "$@"
> do if [ "$arg" = "-f" -o "$arg" = "-e" ]
>then beeline -u "..." "$@"
> return
>fi
> done
> beeline -u "..." "$@" -f /dev/stdin
> }
> beeline_workaround_exit_status < boo;
> EOF
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13039) BETWEEN predicate is not functioning correctly with predicate pushdown on Parquet table
[ https://issues.apache.org/jira/browse/HIVE-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149371#comment-15149371 ] Sergio Peña commented on HIVE-13039: [~ychena] Can we add those unit-tests to branch-1? At least the ones that do not take too much time to include. > BETWEEN predicate is not functioning correctly with predicate pushdown on > Parquet table > --- > > Key: HIVE-13039 > URL: https://issues.apache.org/jira/browse/HIVE-13039 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.2.1, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.1.0 > > Attachments: HIVE-13039.1.branch1.txt, HIVE-13039.1.patch, > HIVE-13039.2.patch, HIVE-13039.3.patch > > > BETWEEN becomes exclusive in parquet table when predicate pushdown is on (as > it is by default in newer Hive versions). To reproduce(in a cluster, not > local setup): > CREATE TABLE parquet_tbl( > key int, > ldate string) > PARTITIONED BY ( > lyear string ) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert overwrite table parquet_tbl partition (lyear='2016') select > 1, > '2016-02-03' from src limit 1; > set hive.optimize.ppd.storage = true; > set hive.optimize.ppd = true; > select * from parquet_tbl where ldate between '2016-02-03' and '2016-02-03'; > No row will be returned in a cluster. > But if you turn off hive.optimize.ppd, one row will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12749) Constant propagate returns string values in incorrect format
[
https://issues.apache.org/jira/browse/HIVE-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149354#comment-15149354
]
Hive QA commented on HIVE-12749:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788045/HIVE-12749.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7003/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7003/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7003/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-7003/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
593a184..6a5a4ae branch-1 -> origin/branch-1
4d67088..06a361c master -> origin/master
+ git reset --hard HEAD
HEAD is now at 4d67088 HIVE-13039: BETWEEN predicate is not functioning
correctly with predicate pushdown on Parquet table (Yongzhi Chen, reviewed by
Sergio Pena)
+ git clean -f -d
Removing
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/SparkJobUtils.java
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 06a361c HIVE-13033 : SPDO unnecessarily duplicates columns in
key & value of mapper output (Ashutosh Chauhan via Prasanth J)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788045 - PreCommit-HIVE-TRUNK-Build
> Constant propagate returns string values in incorrect format
>
>
> Key: HIVE-12749
> URL: https://issues.apache.org/jira/browse/HIVE-12749
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0
>Reporter: Oleksiy Sayankin
>Assignee: Aleksey Vovchenko
> Attachments: HIVE-12749.1.patch, HIVE-12749.2.patch,
> HIVE-12749.3.patch, HIVE-12749.4.patch
>
>
> h2. STEP 1. Create and upload test data
> Execute in command line:
> {noformat}
> nano stest.data
> {noformat}
> Add to file:
> {noformat}
> 000126,000777
> 000126,000778
> 000126,000779
> 000474,000888
> 000468,000889
> 000272,000880
> {noformat}
> {noformat}
> hadoop fs -put stest.data /
> {noformat}
> {noformat}
> hive> create table stest(x STRING, y STRING) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY ',';
> hive> LOAD DATA INPATH '/stest.data' OVERWRITE INTO TABLE stest;
> {noformat}
> h2. STEP 2. Execute test query (with cast for x)
> {noformat}
> select x from stest where cast(x as int) = 126;
> {noformat}
> EXPECTED RESULT:
> {noformat}
> 000126
> 000126
> 000126
> {noformat}
> ACTUAL RESULT:
> {noformat}
> 126
> 126
> 126
> {noformat}
> h2. STEP 3. Execute test query (no cast for x)
> {noformat}
> hive> select x from stest where x
[jira] [Updated] (HIVE-13039) BETWEEN predicate is not functioning correctly with predicate pushdown on Parquet table
[ https://issues.apache.org/jira/browse/HIVE-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-13039: Attachment: HIVE-13039.1.branch1.txt > BETWEEN predicate is not functioning correctly with predicate pushdown on > Parquet table > --- > > Key: HIVE-13039 > URL: https://issues.apache.org/jira/browse/HIVE-13039 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.2.1, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.1.0 > > Attachments: HIVE-13039.1.branch1.txt, HIVE-13039.1.patch, > HIVE-13039.2.patch, HIVE-13039.3.patch > > > BETWEEN becomes exclusive in parquet table when predicate pushdown is on (as > it is by default in newer Hive versions). To reproduce(in a cluster, not > local setup): > CREATE TABLE parquet_tbl( > key int, > ldate string) > PARTITIONED BY ( > lyear string ) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert overwrite table parquet_tbl partition (lyear='2016') select > 1, > '2016-02-03' from src limit 1; > set hive.optimize.ppd.storage = true; > set hive.optimize.ppd = true; > select * from parquet_tbl where ldate between '2016-02-03' and '2016-02-03'; > No row will be returned in a cluster. > But if you turn off hive.optimize.ppd, one row will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13039) BETWEEN predicate is not functioning correctly with predicate pushdown on Parquet table
[ https://issues.apache.org/jira/browse/HIVE-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149348#comment-15149348 ] Yongzhi Chen commented on HIVE-13039: - Thanks [~spena] for reviewing the code. The following 3 files are not in branch-1, so I remove the changes related to the 3 file. ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRecordReaderWrapper.java ql/src/test/org/apache/hadoop/hive/ql/io/parquet/read/TestParquetFilterPredicate.java ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestConvertAstToSearchArg.java attach the change for branch-1. > BETWEEN predicate is not functioning correctly with predicate pushdown on > Parquet table > --- > > Key: HIVE-13039 > URL: https://issues.apache.org/jira/browse/HIVE-13039 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.2.1, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.1.0 > > Attachments: HIVE-13039.1.patch, HIVE-13039.2.patch, > HIVE-13039.3.patch > > > BETWEEN becomes exclusive in parquet table when predicate pushdown is on (as > it is by default in newer Hive versions). To reproduce(in a cluster, not > local setup): > CREATE TABLE parquet_tbl( > key int, > ldate string) > PARTITIONED BY ( > lyear string ) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert overwrite table parquet_tbl partition (lyear='2016') select > 1, > '2016-02-03' from src limit 1; > set hive.optimize.ppd.storage = true; > set hive.optimize.ppd = true; > select * from parquet_tbl where ldate between '2016-02-03' and '2016-02-03'; > No row will be returned in a cluster. > But if you turn off hive.optimize.ppd, one row will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client
[
https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149350#comment-15149350
]
Hive QA commented on HIVE-12205:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788035/HIVE-12205.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9790 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7002/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7002/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7002/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788035 - PreCommit-HIVE-TRUNK-Build
> Spark: unify spark statististics aggregation between local and remote spark
> client
> --
>
> Key: HIVE-12205
> URL: https://issues.apache.org/jira/browse/HIVE-12205
> Project: Hive
> Issue Type: Task
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-12205.1.patch, HIVE-12205.2.patch,
> HIVE-12205.3.patch
>
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark
> statistics aggregation are done similar but in different code paths. Ideally,
> we should have a unified approach to simply maintenance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149320#comment-15149320 ] Ashutosh Chauhan commented on HIVE-12988: - Failures are not related. Patch is ready for review. > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13065) Hive throws NPE when writing map type data to a HBase backed table
[
https://issues.apache.org/jira/browse/HIVE-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13065:
Attachment: HIVE-13065.1.patch
Fix by check null before write separator(key-value) and value.
> Hive throws NPE when writing map type data to a HBase backed table
> --
>
> Key: HIVE-13065
> URL: https://issues.apache.org/jira/browse/HIVE-13065
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13065.1.patch
>
>
> Hive throws NPE when writing data to a HBase backed table with below
> conditions:
> # There is a map type column
> # The map type column has NULL in its values
> Below are the reproduce steps:
> *1) Create a HBase backed Hive table*
> {code:sql}
> create table hbase_test (id bigint, data map)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with serdeproperties ("hbase.columns.mapping" = ":key,cf:map_col")
> tblproperties ("hbase.table.name" = "hive_test");
> {code}
> *2) insert data into above table*
> {code:sql}
> insert overwrite table hbase_test select 1 as id, map('abcd', null) as data
> from src limit 1;
> {code}
> The mapreduce job for insert query fails. Error messages are as below:
> {noformat}
> 2016-02-15 02:26:33,225 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0) {"key":{},"value":{"_col0":1,"_col1":{"abcd":null}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:265)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{},"value":{"_col0":1,"_col1":{"abcd":null}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:253)
> ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:731)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.LimitOperator.processOp(LimitOperator.java:51)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:244)
> ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:286)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:666)
> ... 14 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:221)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:275)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:282)
> ... 15 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149296#comment-15149296 ] Prasanth Jayachandran commented on HIVE-12558: -- Wouldn't that tell something about internal fragmentation in allocator? And accordingly tune the min allocation size.. > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch, > HIVE-12558.wip.patch, sample-output.png > > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13056) delegation tokens do not work with HS2 when used with http transport and kerberos
[
https://issues.apache.org/jira/browse/HIVE-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149293#comment-15149293
]
Thejas M Nair commented on HIVE-13056:
--
+1
> delegation tokens do not work with HS2 when used with http transport and
> kerberos
> -
>
> Key: HIVE-13056
> URL: https://issues.apache.org/jira/browse/HIVE-13056
> Project: Hive
> Issue Type: Bug
> Components: Authentication
>Affects Versions: 1.2.1
>Reporter: Cheng Xu
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Attachments: HIVE-13056.patch
>
>
> We're getting a HiveSQLException on secure windows clusters.
> {code}
> 2016-02-08
> 13:48:09,535|beaver.machine|INFO|6114|140264674350912|MainThread|Job ID :
> 000-160208134528402-oozie-oozi-W
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|Workflow
> Name : hive2-wf
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|App Path
> :
> wasb://oozie1-hb...@humbtestings5jp.blob.core.windows.net/user/hrt_qa/test_hiveserver2
> 2016-02-08
> 13:48:09,536|beaver.machine|INFO|6114|140264674350912|MainThread|Status
> : KILLED
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|Run
> : 0
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|User
> : hrt_qa
> 2016-02-08
> 13:48:09,537|beaver.machine|INFO|6114|140264674350912|MainThread|Group
> : -
> 2016-02-08
> 13:48:09,547|beaver.machine|INFO|6114|140264674350912|MainThread|Created
> : 2016-02-08 13:47 GMT
> 2016-02-08
> 13:48:09,548|beaver.machine|INFO|6114|140264674350912|MainThread|Started
> : 2016-02-08 13:47 GMT
> 2016-02-08
> 13:48:09,552|beaver.machine|INFO|6114|140264674350912|MainThread|Last
> Modified : 2016-02-08 13:48 GMT
> 2016-02-08
> 13:48:09,553|beaver.machine|INFO|6114|140264674350912|MainThread|Ended
> : 2016-02-08 13:48 GMT
> 2016-02-08
> 13:48:09,553|beaver.machine|INFO|6114|140264674350912|MainThread|CoordAction
> ID: -
> 2016-02-08 13:48:09,566|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,566|beaver.machine|INFO|6114|140264674350912|MainThread|Actions
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|ID
> Status
> Ext ID Ext Status Err Code
> 2016-02-08
> 13:48:09,567|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,571|beaver.machine|INFO|6114|140264674350912|MainThread|000-160208134528402-oozie-oozi-W@:start:
> OK- OK
> -
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|000-160208134528402-oozie-oozi-W@hive-node
> ERROR - ERROR
> HiveSQLException
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|000-160208134528402-oozie-oozi-W@fail
> OK- OK
>E0729
> 2016-02-08
> 13:48:09,572|beaver.machine|INFO|6114|140264674350912|MainThread|
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11355) Hive on tez: memory manager for sort buffers (input/output) and operators
[ https://issues.apache.org/jira/browse/HIVE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-11355: -- Attachment: HIVE-11355.17.patch > Hive on tez: memory manager for sort buffers (input/output) and operators > - > > Key: HIVE-11355 > URL: https://issues.apache.org/jira/browse/HIVE-11355 > Project: Hive > Issue Type: Improvement > Components: Tez >Affects Versions: 2.0.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-11355.1.patch, HIVE-11355.10.patch, > HIVE-11355.11.patch, HIVE-11355.12.patch, HIVE-11355.13.patch, > HIVE-11355.14.patch, HIVE-11355.15.patch, HIVE-11355.16.patch, > HIVE-11355.17.patch, HIVE-11355.2.patch, HIVE-11355.3.patch, > HIVE-11355.4.patch, HIVE-11355.5.patch, HIVE-11355.6.patch, > HIVE-11355.7.patch, HIVE-11355.8.patch, HIVE-11355.9.patch > > > We need to better manage the sort buffer allocations to ensure better > performance. Also, we need to provide configurations to certain operators to > stay within memory limits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10308) Vectorization execution throws java.lang.IllegalArgumentException: Unsupported complex type: MAP
[
https://issues.apache.org/jira/browse/HIVE-10308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-10308:
Fix Version/s: 1.2.2
2.0.0
1.3.0
> Vectorization execution throws java.lang.IllegalArgumentException:
> Unsupported complex type: MAP
>
>
> Key: HIVE-10308
> URL: https://issues.apache.org/jira/browse/HIVE-10308
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 0.14.0, 0.13.1, 1.2.0, 1.1.0
>Reporter: Selina Zhang
>Assignee: Matt McCline
> Fix For: 1.3.0, 2.0.0, 1.2.2
>
> Attachments: HIVE-10308.1.patch
>
>
> Steps to reproduce:
> {code:sql}
> CREATE TABLE test_orc (a INT, b MAP) STORED AS ORC;
> INSERT OVERWRITE TABLE test_orc SELECT 1, MAP(1, "one", 2, "two") FROM src
> LIMIT 1;
> CREATE TABLE test(key INT) ;
> INSERT OVERWRITE TABLE test SELECT 1 FROM src LIMIT 1;
> set hive.vectorized.execution.enabled=true;
> set hive.auto.convert.join=false;
> select l.key from test l left outer join test_orc r on (l.key= r.a) where r.a
> is not null;
> {code}
> Stack trace:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Unsupported complex type: MAP
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.genVectorExpressionWritable(VectorExpressionWriterFactory.java:456)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.processVectorInspector(VectorExpressionWriterFactory.java:1191)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.initializeOp(VectorReduceSinkOperator.java:58)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:442)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:198)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12612) beeline always exits with 0 status when reading query from standard input
[
https://issues.apache.org/jira/browse/HIVE-12612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149205#comment-15149205
]
Paulo Sequeira commented on HIVE-12612:
---
Have you tried feeding Beeline with an invalid statement? The issue is that, if
the query is malformed or fails for some other reason, the exit status of the
Beeline process is still 0 (which indicates that the command completed
successfully) rather than the expected non-zero status indicating that there
was a failure.
Note that it will be evident from what Beeline prints on STDOUT and STDERR that
it detected the failure, but if you use the shell commands that test the actual
exit status (like the 'if' statement or the '&&' operator), then they would
behave as if the command ran successfully.
> beeline always exits with 0 status when reading query from standard input
> -
>
> Key: HIVE-12612
> URL: https://issues.apache.org/jira/browse/HIVE-12612
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
> Environment: CDH5.5.0
>Reporter: Paulo Sequeira
>Priority: Minor
>
> Similar to what was reported on HIVE-6978, but now it only happens when the
> query is read from the standard input. For example, the following fails as
> expected:
> {code}
> bash$ if beeline -u "jdbc:hive2://..." -e "boo;" ; then echo "Ok?!" ; else
> echo "Failed!" ; fi
> Connecting to jdbc:hive2://...
> Connected to: Apache Hive (version 1.1.0-cdh5.5.0)
> Driver: Hive JDBC (version 1.1.0-cdh5.5.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:0
> cannot recognize input near 'boo' '' '' (state=42000,code=4)
> Closing: 0: jdbc:hive2://...
> Failed!
> {code}
> But the following does not:
> {code}
> bash$ if echo "boo;"|beeline -u "jdbc:hive2://..." ; then echo "Ok?!" ; else
> echo "Failed!" ; fi
> Connecting to jdbc:hive2://...
> Connected to: Apache Hive (version 1.1.0-cdh5.5.0)
> Driver: Hive JDBC (version 1.1.0-cdh5.5.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 1.1.0-cdh5.5.0 by Apache Hive
> 0: jdbc:hive2://...:8> Error: Error while compiling statement: FAILED:
> ParseException line 1:0 cannot recognize input near 'boo' '' ''
> (state=42000,code=4)
> 0: jdbc:hive2://...:8> Closing: 0: jdbc:hive2://...
> Ok?!
> {code}
> This was misleading our batch scripts to always believe that the execution of
> the queries succeded, when sometimes that was not the case.
> h2. Workaround
> We found we can work around the issue by always using the -e or the -f
> parameters, and even reading the standard input through the /dev/stdin device
> (this was useful because a lot of the scripts fed the queries from here
> documents), like this:
> {code:title=some-script.sh}
> #!/bin/sh
> set -o nounset -o errexit -o pipefail
> # As beeline is failing to report an error status if reading the query
> # to be executed from STDIN, check whether no -f or -e option is used
> # and, in that case, pretend it has to read the query from a regular
> # file using -f to read from /dev/stdin
> function beeline_workaround_exit_status () {
> for arg in "$@"
> do if [ "$arg" = "-f" -o "$arg" = "-e" ]
>then beeline -u "..." "$@"
> return
>fi
> done
> beeline -u "..." "$@" -f /dev/stdin
> }
> beeline_workaround_exit_status < boo;
> EOF
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13062) Hive metastore test failing
[
https://issues.apache.org/jira/browse/HIVE-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149148#comment-15149148
]
Jesus Camacho Rodriguez commented on HIVE-13062:
Pushed to master, thanks [~sershe]!
> Hive metastore test failing
> ---
>
> Key: HIVE-13062
> URL: https://issues.apache.org/jira/browse/HIVE-13062
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 2.1.0
>
> Attachments: HIVE-13062.patch
>
>
> The testutils/metastore/metastore-upgrade-test.sh script takes the list of
> scripts to execute for the DB upgrade from the folder itself
> ({{metastore/scripts/upgrade/DB_NAME}}). Currently it fails as it executes
> upgrade-1.2.0-to-1.3.0 followed by upgrade-1.2.0-to-2.0.0.
> Instead, it should use the content of the {{upgrade.order.DB_NAME}} file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12612) beeline always exits with 0 status when reading query from standard input
[
https://issues.apache.org/jira/browse/HIVE-12612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149142#comment-15149142
]
Ruslan Dautkhanov commented on HIVE-12612:
--
Just tested, it works for me (I have bl as an alias for beeline to connect to
one of the clusters):
{code}
$ echo "show databases;" | bl
scan complete in 4ms
Connecting to jdbc:hive2://xx.yy:1/default;principal=hive/_HOST@HADOOP
Connected to: Apache Hive (version 1.1.0-cdh5.5.1)
Driver: Hive JDBC (version 1.1.0-cdh5.5.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.1.0-cdh5.5.1 by Apache Hive
0: jdbc:hive2://xx.yy.com:1> show databases;
+---+--+
| database_name |
+---+--+
| default |
. . .
+---+--+
13 rows selected (0.832 seconds)
0: jdbc:hive2://xx.yy.com:1> Closing: 0: jdbc:hive2://. . .
rdautkha $
{code}
> beeline always exits with 0 status when reading query from standard input
> -
>
> Key: HIVE-12612
> URL: https://issues.apache.org/jira/browse/HIVE-12612
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
> Environment: CDH5.5.0
>Reporter: Paulo Sequeira
>Priority: Minor
>
> Similar to what was reported on HIVE-6978, but now it only happens when the
> query is read from the standard input. For example, the following fails as
> expected:
> {code}
> bash$ if beeline -u "jdbc:hive2://..." -e "boo;" ; then echo "Ok?!" ; else
> echo "Failed!" ; fi
> Connecting to jdbc:hive2://...
> Connected to: Apache Hive (version 1.1.0-cdh5.5.0)
> Driver: Hive JDBC (version 1.1.0-cdh5.5.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:0
> cannot recognize input near 'boo' '' '' (state=42000,code=4)
> Closing: 0: jdbc:hive2://...
> Failed!
> {code}
> But the following does not:
> {code}
> bash$ if echo "boo;"|beeline -u "jdbc:hive2://..." ; then echo "Ok?!" ; else
> echo "Failed!" ; fi
> Connecting to jdbc:hive2://...
> Connected to: Apache Hive (version 1.1.0-cdh5.5.0)
> Driver: Hive JDBC (version 1.1.0-cdh5.5.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 1.1.0-cdh5.5.0 by Apache Hive
> 0: jdbc:hive2://...:8> Error: Error while compiling statement: FAILED:
> ParseException line 1:0 cannot recognize input near 'boo' '' ''
> (state=42000,code=4)
> 0: jdbc:hive2://...:8> Closing: 0: jdbc:hive2://...
> Ok?!
> {code}
> This was misleading our batch scripts to always believe that the execution of
> the queries succeded, when sometimes that was not the case.
> h2. Workaround
> We found we can work around the issue by always using the -e or the -f
> parameters, and even reading the standard input through the /dev/stdin device
> (this was useful because a lot of the scripts fed the queries from here
> documents), like this:
> {code:title=some-script.sh}
> #!/bin/sh
> set -o nounset -o errexit -o pipefail
> # As beeline is failing to report an error status if reading the query
> # to be executed from STDIN, check whether no -f or -e option is used
> # and, in that case, pretend it has to read the query from a regular
> # file using -f to read from /dev/stdin
> function beeline_workaround_exit_status () {
> for arg in "$@"
> do if [ "$arg" = "-f" -o "$arg" = "-e" ]
>then beeline -u "..." "$@"
> return
>fi
> done
> beeline -u "..." "$@" -f /dev/stdin
> }
> beeline_workaround_exit_status < boo;
> EOF
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9457) Fix obsolete parameter name in HiveConf description of hive.hashtable.initialCapacity
[
https://issues.apache.org/jira/browse/HIVE-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149131#comment-15149131
]
Shannon Ladymon commented on HIVE-9457:
---
The test failures do not seem to be related to this patch. [~sershe], could
you check and review?
> Fix obsolete parameter name in HiveConf description of
> hive.hashtable.initialCapacity
> -
>
> Key: HIVE-9457
> URL: https://issues.apache.org/jira/browse/HIVE-9457
> Project: Hive
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 0.14.0
>Reporter: Lefty Leverenz
>Assignee: Shannon Ladymon
>Priority: Minor
> Attachments: HIVE-9457.patch
>
>
> The description of *hive.hashtable.initialCapacity* in HiveConf.java refers
> to a parameter that existed in an early patch for HIVE-7616
> ("hive.hashtable.stats.key.estimate.adjustment") but was renamed in later
> patches. So change *hive.hashtable.stats.key.estimate.adjustment* to
> *hive.hashtable.key.count.adjustment* in this parameter definition in
> HiveConf.java:
> {code}
> HIVEHASHTABLETHRESHOLD("hive.hashtable.initialCapacity", 10, "Initial
> capacity of " +
> "mapjoin hashtable if statistics are absent, or if
> hive.hashtable.stats.key.estimate.adjustment is set to 0"),
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13062) Hive metastore test failing
[
https://issues.apache.org/jira/browse/HIVE-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149115#comment-15149115
]
Sergey Shelukhin commented on HIVE-13062:
-
+1
> Hive metastore test failing
> ---
>
> Key: HIVE-13062
> URL: https://issues.apache.org/jira/browse/HIVE-13062
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13062.patch
>
>
> The testutils/metastore/metastore-upgrade-test.sh script takes the list of
> scripts to execute for the DB upgrade from the folder itself
> ({{metastore/scripts/upgrade/DB_NAME}}). Currently it fails as it executes
> upgrade-1.2.0-to-1.3.0 followed by upgrade-1.2.0-to-2.0.0.
> Instead, it should use the content of the {{upgrade.order.DB_NAME}} file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13058) Add session and operation_log directory deletion messages
[
https://issues.apache.org/jira/browse/HIVE-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149114#comment-15149114
]
Ashutosh Chauhan commented on HIVE-13058:
-
+1
> Add session and operation_log directory deletion messages
> -
>
> Key: HIVE-13058
> URL: https://issues.apache.org/jira/browse/HIVE-13058
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HIVE-13058.1.patch
>
>
> In the current master, when connecting to HS2, we can confirm that some
> directories were created. For example the following messages of HS2 log.
> {code}
> 2016-02-14T13:57:52,412 INFO [HiveServer2-Handler-Pool: Thread-607]:
> session.SessionState (SessionState.java:createPath(693)) - Created HDFS
> directory: /tmp/hive/root/bfe28a51-5804-4cbb-952c-26709aed4652
> 2016-02-14T13:57:52,416 INFO [HiveServer2-Handler-Pool: Thread-607]:
> session.SessionState (SessionState.java:createPath(693)) - Created local
> directory: /tmp/hive/bfe28a51-5804-4cbb-952c-26709aed4652
> 2016-02-14T13:57:52,419 INFO [HiveServer2-Handler-Pool: Thread-607]:
> session.SessionState (SessionState.java:createPath(693)) - Created HDFS
> directory: /tmp/hive/root/bfe28a51-5804-4cbb-952c-26709aed4652/_tmp_space.db
> 2016-02-14T13:57:52,421 INFO [HiveServer2-Handler-Pool: Thread-607]:
> session.HiveSessionImpl (HiveSessionImpl.java:setOperationLogSessionDir(287))
> - Operation log session directory is created:
> /tmp/hive/operation_logs/bfe28a51-5804-4cbb-952c-26709aed4652
> {code}
> But when closing from HS2, we cannot confirm that these directories were
> deleted. So I change it so that some messages about these directories
> deletion may be output as follows.
> {code}
> 2016-02-14T13:57:56,877 INFO
> [bfe28a51-5804-4cbb-952c-26709aed4652bfe28a51-5804-4cbb-952c-26709aed4652HiveServer2-Handler-Pool:
> Thread-607]: session.HiveSessionImpl
> (HiveSessionImpl.java:cleanupSessionLogDir(690)) - Operation log session
> directory is deleted:
> /tmp/hive/operation_logs/bfe28a51-5804-4cbb-952c-26709aed4652
> 2016-02-14T13:57:56,890 INFO
> [bfe28a51-5804-4cbb-952c-26709aed4652bfe28a51-5804-4cbb-952c-26709aed4652HiveServer2-Handler-Pool:
> Thread-607]: session.SessionState (SessionState.java:dropSessionPaths(750))
> - Deleted HDFS directory: /tmp/hive/root/bfe28a51-5804-4cbb-952c-26709aed4652
> 2016-02-14T13:57:56,891 INFO
> [bfe28a51-5804-4cbb-952c-26709aed4652bfe28a51-5804-4cbb-952c-26709aed4652HiveServer2-Handler-Pool:
> Thread-607]: session.SessionState (SessionState.java:dropSessionPaths(754))
> - Deleted local directory: /tmp/hive/bfe28a51-5804-4cbb-952c-26709aed4652
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13015) Bundle Log4j2 jars with hive-exec
[
https://issues.apache.org/jira/browse/HIVE-13015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149075#comment-15149075
]
Prasanth Jayachandran commented on HIVE-13015:
--
Weirdly, I ran couple of these test locally and they seem to pass without any
issues.
> Bundle Log4j2 jars with hive-exec
> -
>
> Key: HIVE-13015
> URL: https://issues.apache.org/jira/browse/HIVE-13015
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Gopal V
> Attachments: HIVE-13015.1.patch, HIVE-13015.1.patch
>
>
> In some of the recent test runs, we are seeing multiple bindings for SLF4j
> that causes issues with LOG4j2 logger.
> {code}
> SLF4J: Found binding in
> [jar:file:/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1454694331819_0001/container_e06_1454694331819_0001_01_02/app/install/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> {code}
> We have added explicit exclusions for slf4j-log4j12 but some library is
> pulling it transitively and it's getting packaged with hive libs. Also hive
> currently uses version 1.7.5 for slf4j. We should add dependency convergence
> for sl4fj and also remove packaging of slf4j-log4j12.*.jar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13015) Bundle Log4j2 jars with hive-exec
[
https://issues.apache.org/jira/browse/HIVE-13015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13015:
-
Attachment: HIVE-13015.2.patch
Lost the test logs. Retying again as these test pass locally for me.
> Bundle Log4j2 jars with hive-exec
> -
>
> Key: HIVE-13015
> URL: https://issues.apache.org/jira/browse/HIVE-13015
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Gopal V
> Attachments: HIVE-13015.1.patch, HIVE-13015.1.patch,
> HIVE-13015.2.patch
>
>
> In some of the recent test runs, we are seeing multiple bindings for SLF4j
> that causes issues with LOG4j2 logger.
> {code}
> SLF4J: Found binding in
> [jar:file:/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1454694331819_0001/container_e06_1454694331819_0001_01_02/app/install/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> {code}
> We have added explicit exclusions for slf4j-log4j12 but some library is
> pulling it transitively and it's getting packaged with hive libs. Also hive
> currently uses version 1.7.5 for slf4j. We should add dependency convergence
> for sl4fj and also remove packaging of slf4j-log4j12.*.jar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13015) Bundle Log4j2 jars with hive-exec
[
https://issues.apache.org/jira/browse/HIVE-13015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-13015:
Assignee: Prasanth Jayachandran (was: Gopal V)
> Bundle Log4j2 jars with hive-exec
> -
>
> Key: HIVE-13015
> URL: https://issues.apache.org/jira/browse/HIVE-13015
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13015.1.patch, HIVE-13015.1.patch,
> HIVE-13015.2.patch
>
>
> In some of the recent test runs, we are seeing multiple bindings for SLF4j
> that causes issues with LOG4j2 logger.
> {code}
> SLF4J: Found binding in
> [jar:file:/grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1454694331819_0001/container_e06_1454694331819_0001_01_02/app/install/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> {code}
> We have added explicit exclusions for slf4j-log4j12 but some library is
> pulling it transitively and it's getting packaged with hive libs. Also hive
> currently uses version 1.7.5 for slf4j. We should add dependency convergence
> for sl4fj and also remove packaging of slf4j-log4j12.*.jar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9457) Fix obsolete parameter name in HiveConf description of hive.hashtable.initialCapacity
[
https://issues.apache.org/jira/browse/HIVE-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149048#comment-15149048
]
Hive QA commented on HIVE-9457:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12787994/HIVE-9457.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 9789 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_create_merge_compressed
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapreduce2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_inner_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_left_outer_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_leftsemi_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_struct_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_12
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_nested_mapjoin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarDataNucleusUnCaching
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7001/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7001/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7001/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12787994 - PreCommit-HIVE-TRUNK-Build
> Fix obsolete parameter name in HiveConf description of
> hive.hashtable.initialCapacity
> -
>
> Key: HIVE-9457
> URL: https://issues.apache.org/jira/browse/HIVE-9457
> Project: Hive
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 0.14.0
>Reporter: Lefty Leverenz
>Assignee: Shannon Ladymon
>Priority: Minor
> Attachments: HIVE-9457.patch
>
>
> The description of *hive.hashtable.initialCapacity* in HiveConf.java refers
> to a parameter that existed in an early patch for HIVE-7616
> ("hive.hashtable.stats.key.estimate.adjustment") but was renamed in later
> patches. So change *hive.hashtable.stats.key.estimate.adjustment* to
> *hive.hashtable.key.count.adjustment* in this parameter definition in
> HiveConf.java:
> {code}
> HIVEHASHTABLETHRESHOLD("hive.hashtable.initialCapacity", 10, "Initial
> capacity of " +
> "mapjoin hashtable if statistics are absent, or if
> hive.hashtable.stats.key.estimate.adjustment is set to 0"),
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12420) WebHCat server throws NPE when you run command with -d user.name.
[
https://issues.apache.org/jira/browse/HIVE-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-12420:
-
Assignee: (was: Thejas M Nair)
> WebHCat server throws NPE when you run command with -d user.name.
> -
>
> Key: HIVE-12420
> URL: https://issues.apache.org/jira/browse/HIVE-12420
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 2.0.0
>Reporter: Takahiko Saito
>Priority: Blocker
>
> When you run with '-d user.name', it failed with:
> {noformat}
> [hrt_qa@os-r6-bccslu-hive-1-r-5 ~]$ curl -s -d user.name=hrt_qa -d
> execute="drop table if exists templetontest_tab2;"
> http://os-r6-bccslu-hive-1-r-3.novalocal:20111/templeton/v1/ddl
>
>
>
> Error 500 Server Error
>
>
> HTTP ERROR: 500
> Problem accessing /templeton/v1/ddl. Reason:
> Server Error
> Powered by Jetty://
>
>
> {noformat}
> server log shows:
> {noformat}
> WARN | 16 Nov 2015 19:48:22,738 | org.eclipse.jetty.servlet.ServletHandler |
> /templeton/v1/ddl
> java.lang.NullPointerException
> at
> org.apache.http.client.utils.URLEncodedUtils.parse(URLEncodedUtils.java:235)
> ~[hive-jdbc-1.2.1.2.3.5.0-13-standalone.jar:1.2.1.2.3.5.0-13]
> at
> org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.getUserName(PseudoAuthenticationHandler.java:143)
> ~[hadoop-auth-2.6.0.jar:?]
> at
> org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:179)
> ~[hadoop-auth-2.6.0.jar:?]
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
> ~[hadoop-auth-2.6.0.jar:?]
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:88)
> ~[hadoop-hdfs-2.7.1.2.3.5.0-13.jar:?]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
> ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at org.eclipse.jetty.server.Server.handle(Server.java:349)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
> WARN | 16 Nov 2015 19:48:22,738 | org.eclipse.jetty.servlet.ServletHandler |
> /templeton/v1/ddl
> java.lang.NullPointerException
> at
> org.apache.http.client.utils.URLEncodedUtils.parse(URLEncodedUtils.java:235)
> ~[hive-jdbc-1.2.1.2.3.5.0-13-standalone.jar:1.2.1.2.3.5.0-13]
> at
>
[jira] [Updated] (HIVE-13049) Hive Thrift python lib not at latest version
[ https://issues.apache.org/jira/browse/HIVE-13049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-13049: - Assignee: (was: Thejas M Nair) > Hive Thrift python lib not at latest version > > > Key: HIVE-13049 > URL: https://issues.apache.org/jira/browse/HIVE-13049 > Project: Hive > Issue Type: Bug > Components: Authentication, Security >Reporter: Sailesh Mukil >Priority: Blocker > > The Thrift python lib in Hive does not seem to be of the latest Thrift > version. We found this while running tests with SSL on our cluster. We have a > python script that runs our workloads and we couldn't get SSL authentication. > Looking at the Hive python thrift library, I noticed that the TSSLSocket.py > file is missing: > https://github.com/apache/hive/tree/master/service/lib/py/thrift/transport > From the thrift repo, we need TSSLSocket.py: > https://github.com/apache/thrift/tree/master/lib/py/src/transport -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13039) BETWEEN predicate is not functioning correctly with predicate pushdown on Parquet table
[ https://issues.apache.org/jira/browse/HIVE-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13039: --- Fix Version/s: 2.1.0 > BETWEEN predicate is not functioning correctly with predicate pushdown on > Parquet table > --- > > Key: HIVE-13039 > URL: https://issues.apache.org/jira/browse/HIVE-13039 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.2.1, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.1.0 > > Attachments: HIVE-13039.1.patch, HIVE-13039.2.patch, > HIVE-13039.3.patch > > > BETWEEN becomes exclusive in parquet table when predicate pushdown is on (as > it is by default in newer Hive versions). To reproduce(in a cluster, not > local setup): > CREATE TABLE parquet_tbl( > key int, > ldate string) > PARTITIONED BY ( > lyear string ) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert overwrite table parquet_tbl partition (lyear='2016') select > 1, > '2016-02-03' from src limit 1; > set hive.optimize.ppd.storage = true; > set hive.optimize.ppd = true; > select * from parquet_tbl where ldate between '2016-02-03' and '2016-02-03'; > No row will be returned in a cluster. > But if you turn off hive.optimize.ppd, one row will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13055) Add unit tests for HIVE-11512
[ https://issues.apache.org/jira/browse/HIVE-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148839#comment-15148839 ] Aihua Xu commented on HIVE-13055: - +1. > Add unit tests for HIVE-11512 > - > > Key: HIVE-13055 > URL: https://issues.apache.org/jira/browse/HIVE-13055 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Affects Versions: 1.3.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13055.patch > > > HIVE-11866 adds the ability to test LDAP authenticator with a LDAP Server. We > need more test coverage for feature in HIVE-11512. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13039) BETWEEN predicate is not functioning correctly with predicate pushdown on Parquet table
[ https://issues.apache.org/jira/browse/HIVE-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148812#comment-15148812 ] Sergio Peña commented on HIVE-13039: Looks good +1 > BETWEEN predicate is not functioning correctly with predicate pushdown on > Parquet table > --- > > Key: HIVE-13039 > URL: https://issues.apache.org/jira/browse/HIVE-13039 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.2.1, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-13039.1.patch, HIVE-13039.2.patch, > HIVE-13039.3.patch > > > BETWEEN becomes exclusive in parquet table when predicate pushdown is on (as > it is by default in newer Hive versions). To reproduce(in a cluster, not > local setup): > CREATE TABLE parquet_tbl( > key int, > ldate string) > PARTITIONED BY ( > lyear string ) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert overwrite table parquet_tbl partition (lyear='2016') select > 1, > '2016-02-03' from src limit 1; > set hive.optimize.ppd.storage = true; > set hive.optimize.ppd = true; > select * from parquet_tbl where ldate between '2016-02-03' and '2016-02-03'; > No row will be returned in a cluster. > But if you turn off hive.optimize.ppd, one row will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11160: --- Description: Hive will collect table stats when set hive.stats.autogather=true during the INSERT OVERWRITE command. And then the users need to collect the column stats themselves using "Analyze" command. In this patch, the column stats will also be collected automatically. More specifically, INSERT OVERWRITE will automatically create new column stats. INSERT INTO will automatically merge new column stats with existing ones. (was: Hive will collect table stats when set hive.stats.autogather=true during the INSERT OVERWRITE command. And then the users need to collect the column stats themselves using "Analyze" command. In this patch, the column stats will also be collected automatically.) > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11160: --- Attachment: HIVE-11160.04.patch > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13039) BETWEEN predicate is not functioning correctly with predicate pushdown on Parquet table
[ https://issues.apache.org/jira/browse/HIVE-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148645#comment-15148645 ] Yongzhi Chen commented on HIVE-13039: - The failures are not related. [~spena], could you review the change? > BETWEEN predicate is not functioning correctly with predicate pushdown on > Parquet table > --- > > Key: HIVE-13039 > URL: https://issues.apache.org/jira/browse/HIVE-13039 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.2.1, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-13039.1.patch, HIVE-13039.2.patch, > HIVE-13039.3.patch > > > BETWEEN becomes exclusive in parquet table when predicate pushdown is on (as > it is by default in newer Hive versions). To reproduce(in a cluster, not > local setup): > CREATE TABLE parquet_tbl( > key int, > ldate string) > PARTITIONED BY ( > lyear string ) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert overwrite table parquet_tbl partition (lyear='2016') select > 1, > '2016-02-03' from src limit 1; > set hive.optimize.ppd.storage = true; > set hive.optimize.ppd = true; > select * from parquet_tbl where ldate between '2016-02-03' and '2016-02-03'; > No row will be returned in a cluster. > But if you turn off hive.optimize.ppd, one row will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12749) Constant propagate returns string values in incorrect format
[
https://issues.apache.org/jira/browse/HIVE-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Vovchenko updated HIVE-12749:
-
Attachment: HIVE-12749.4.patch
> Constant propagate returns string values in incorrect format
>
>
> Key: HIVE-12749
> URL: https://issues.apache.org/jira/browse/HIVE-12749
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0
>Reporter: Oleksiy Sayankin
>Assignee: Aleksey Vovchenko
> Attachments: HIVE-12749.1.patch, HIVE-12749.2.patch,
> HIVE-12749.3.patch, HIVE-12749.4.patch
>
>
> h2. STEP 1. Create and upload test data
> Execute in command line:
> {noformat}
> nano stest.data
> {noformat}
> Add to file:
> {noformat}
> 000126,000777
> 000126,000778
> 000126,000779
> 000474,000888
> 000468,000889
> 000272,000880
> {noformat}
> {noformat}
> hadoop fs -put stest.data /
> {noformat}
> {noformat}
> hive> create table stest(x STRING, y STRING) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY ',';
> hive> LOAD DATA INPATH '/stest.data' OVERWRITE INTO TABLE stest;
> {noformat}
> h2. STEP 2. Execute test query (with cast for x)
> {noformat}
> select x from stest where cast(x as int) = 126;
> {noformat}
> EXPECTED RESULT:
> {noformat}
> 000126
> 000126
> 000126
> {noformat}
> ACTUAL RESULT:
> {noformat}
> 126
> 126
> 126
> {noformat}
> h2. STEP 3. Execute test query (no cast for x)
> {noformat}
> hive> select x from stest where x = 126;
> {noformat}
> EXPECTED RESULT:
> {noformat}
> 000126
> 000126
> 000126
> {noformat}
> ACTUAL RESULT:
> {noformat}
> 126
> 126
> 126
> {noformat}
> In steps #2, #3 I expected '000126' because the origin type of x is STRING in
> stest table.
> Note, setting hive.optimize.constant.propagation=false fixes the issue.
> {noformat}
> hive> set hive.optimize.constant.propagation=false;
> hive> select x from stest where x = 126;
> OK
> 000126
> 000126
> 000126
> {noformat}
> Related to HIVE-11104, HIVE-8555
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8339) Job status not found after 100% succeded map
[ https://issues.apache.org/jira/browse/HIVE-8339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengzhaozheng updated HIVE-8339: Assignee: (was: zengzhaozheng) > Job status not found after 100% succeded map > --- > > Key: HIVE-8339 > URL: https://issues.apache.org/jira/browse/HIVE-8339 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 > Environment: Hadoop 2.4.0, Hive 0.13.1. > Amazon EMR cluster of 9 i2.4xlarge nodes. > 800+GB of data in HDFS. >Reporter: Valera Chevtaev > > According to the logs it seems that the jobs 100% succeed for both map and > reduce but then wasn't able to get the status of the job from job history > server. > Hive logs: > 2014-10-03 07:57:26,593 INFO [main]: exec.Task > (SessionState.java:printInfo(536)) - 2014-10-03 07:57:26,593 Stage-1 map = > 100%, reduce = 99%, Cumulative CPU 872541.02 sec > 2014-10-03 07:57:47,447 INFO [main]: exec.Task > (SessionState.java:printInfo(536)) - 2014-10-03 07:57:47,446 Stage-1 map = > 100%, reduce = 100%, Cumulative CPU 872566.55 sec > 2014-10-03 07:57:48,710 INFO [main]: mapred.ClientServiceDelegate > (ClientServiceDelegate.java:getProxy(273)) - Application state is completed. > FinalApplicationStatus=SUCCEEDED. Redirecting to job history server > 2014-10-03 07:57:48,716 ERROR [main]: exec.Task > (SessionState.java:printError(545)) - Ended Job = job_1412263771568_0002 with > exception 'java.io.IOException(Could not find status of > job:job_1412263771568_0002)' > java.io.IOException: Could not find status of job:job_1412263771568_0002 >at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:294) >at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:547) >at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:426) >at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136) >at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) >at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) >at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503) >at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270) >at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088) >at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) >at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) >at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:275) >at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:227) >at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:430) >at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:366) >at > org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:463) >at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:479) >at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:759) >at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:697) >at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:636) >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >at java.lang.reflect.Method.invoke(Method.java:606) >at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > 2014-10-03 07:57:48,763 ERROR [main]: ql.Driver > (SessionState.java:printError(545)) - FAILED: Execution Error, return code 1 > from org.apache.hadoop.hive.ql.exec.mr.MapRedTask -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12612) beeline always exits with 0 status when reading query from standard input
[
https://issues.apache.org/jira/browse/HIVE-12612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148536#comment-15148536
]
Paulo Sequeira commented on HIVE-12612:
---
Ruslan, thank you for looking into this. In order to reproduce the issue, you
don't have to read from /dev/stdin (that was in fact used as part of the
workaround instead).
To reproduce, type instead the following in a bash shell:
{code}
if echo "boo;" | beeline -u "jdbc:hive2://..."
then echo 'This is a bug!'
else echo 'Failed as expected'
fi
{code}
> beeline always exits with 0 status when reading query from standard input
> -
>
> Key: HIVE-12612
> URL: https://issues.apache.org/jira/browse/HIVE-12612
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
> Environment: CDH5.5.0
>Reporter: Paulo Sequeira
>Priority: Minor
>
> Similar to what was reported on HIVE-6978, but now it only happens when the
> query is read from the standard input. For example, the following fails as
> expected:
> {code}
> bash$ if beeline -u "jdbc:hive2://..." -e "boo;" ; then echo "Ok?!" ; else
> echo "Failed!" ; fi
> Connecting to jdbc:hive2://...
> Connected to: Apache Hive (version 1.1.0-cdh5.5.0)
> Driver: Hive JDBC (version 1.1.0-cdh5.5.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error while compiling statement: FAILED: ParseException line 1:0
> cannot recognize input near 'boo' '' '' (state=42000,code=4)
> Closing: 0: jdbc:hive2://...
> Failed!
> {code}
> But the following does not:
> {code}
> bash$ if echo "boo;"|beeline -u "jdbc:hive2://..." ; then echo "Ok?!" ; else
> echo "Failed!" ; fi
> Connecting to jdbc:hive2://...
> Connected to: Apache Hive (version 1.1.0-cdh5.5.0)
> Driver: Hive JDBC (version 1.1.0-cdh5.5.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 1.1.0-cdh5.5.0 by Apache Hive
> 0: jdbc:hive2://...:8> Error: Error while compiling statement: FAILED:
> ParseException line 1:0 cannot recognize input near 'boo' '' ''
> (state=42000,code=4)
> 0: jdbc:hive2://...:8> Closing: 0: jdbc:hive2://...
> Ok?!
> {code}
> This was misleading our batch scripts to always believe that the execution of
> the queries succeded, when sometimes that was not the case.
> h2. Workaround
> We found we can work around the issue by always using the -e or the -f
> parameters, and even reading the standard input through the /dev/stdin device
> (this was useful because a lot of the scripts fed the queries from here
> documents), like this:
> {code:title=some-script.sh}
> #!/bin/sh
> set -o nounset -o errexit -o pipefail
> # As beeline is failing to report an error status if reading the query
> # to be executed from STDIN, check whether no -f or -e option is used
> # and, in that case, pretend it has to read the query from a regular
> # file using -f to read from /dev/stdin
> function beeline_workaround_exit_status () {
> for arg in "$@"
> do if [ "$arg" = "-f" -o "$arg" = "-e" ]
>then beeline -u "..." "$@"
> return
>fi
> done
> beeline -u "..." "$@" -f /dev/stdin
> }
> beeline_workaround_exit_status < boo;
> EOF
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8214) Release 0.13.1 missing hwi-war file
[ https://issues.apache.org/jira/browse/HIVE-8214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148500#comment-15148500 ] Hanish Bansal commented on HIVE-8214: - Any update on this one? Does hive still supports hwi, If yes then how to build or get hwi.war file? > Release 0.13.1 missing hwi-war file > --- > > Key: HIVE-8214 > URL: https://issues.apache.org/jira/browse/HIVE-8214 > Project: Hive > Issue Type: Bug > Components: Web UI >Affects Versions: 0.13.1 >Reporter: Naimdjon Takhirov >Priority: Minor > Labels: HIVE-8214.1.patch, branch-0.14, trunk > Attachments: HIVE-8214.1.patch, HIVE-8214.2.patch > > > Starting the Hive with --service hwi option: > $opt/hive/latest: hive --service hwi > ls: /opt/hive/latest/lib/hive-hwi-*.war: No such file or directory > 14/09/22 11:43:46 INFO hwi.HWIServer: HWI is starting up > 14/09/22 11:43:46 INFO mortbay.log: Logging to > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > org.mortbay.log.Slf4jLog > 14/09/22 11:43:46 INFO mortbay.log: jetty-6.1.26 > 14/09/22 11:43:47 INFO mortbay.log: Started SocketConnector@0.0.0.0: > When navigating to localhost:, it just shows the directory index. Looking > at the distribution, the war file is missing in the lib directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13064) Serde properties are not working while expecting output in a directory
[ https://issues.apache.org/jira/browse/HIVE-13064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148481#comment-15148481 ] Rajat Khandelwal commented on HIVE-13064: - Created https://reviews.apache.org/r/43597/ > Serde properties are not working while expecting output in a directory > -- > > Key: HIVE-13064 > URL: https://issues.apache.org/jira/browse/HIVE-13064 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12673) Orcfiledump throws NPE when no files are available
[
https://issues.apache.org/jira/browse/HIVE-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148474#comment-15148474
]
Hive QA commented on HIVE-12673:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12787981/HIVE-12673.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9772 tests executed
*Failed tests:*
{noformat}
TestHBaseNegativeCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-orc_vectorization_ppd.q-vector_left_outer_join2.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6999/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6999/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6999/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12787981 - PreCommit-HIVE-TRUNK-Build
> Orcfiledump throws NPE when no files are available
> --
>
> Key: HIVE-12673
> URL: https://issues.apache.org/jira/browse/HIVE-12673
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-12673.1.patch, HIVE-12673.2.patch,
> HIVE-12673.2.patch, HIVE-12673.3.patch, HIVE-12673.3.rebased.patch
>
>
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.codehaus.jettison.json.JSONTokener.more(JSONTokener.java:106)
> at org.codehaus.jettison.json.JSONTokener.next(JSONTokener.java:116)
> at
> org.codehaus.jettison.json.JSONTokener.nextClean(JSONTokener.java:170)
> at org.codehaus.jettison.json.JSONObject.(JSONObject.java:185)
> at org.codehaus.jettison.json.JSONObject.(JSONObject.java:293)
> at
> org.apache.hadoop.hive.ql.io.orc.JsonFileDump.printJsonMetaData(JsonFileDump.java:197)
> at org.apache.hadoop.hive.ql.io.orc.FileDump.main(FileDump.java:107)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> hive --orcfiledump -j -p /tmp/orc/inventory/inv_date_sk=2452654
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13064) Serde properties are not working while expecting output in a directory
[ https://issues.apache.org/jira/browse/HIVE-13064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148461#comment-15148461 ] Rajat Khandelwal commented on HIVE-13064: - Looking at https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/insert_overwrite_directory.q.out, it seems that despite providing serialization.null.format'='-', nulls are still coming as \N in the result. > Serde properties are not working while expecting output in a directory > -- > > Key: HIVE-13064 > URL: https://issues.apache.org/jira/browse/HIVE-13064 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client
[
https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-12205:
Attachment: HIVE-12205.3.patch
> Spark: unify spark statististics aggregation between local and remote spark
> client
> --
>
> Key: HIVE-12205
> URL: https://issues.apache.org/jira/browse/HIVE-12205
> Project: Hive
> Issue Type: Task
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-12205.1.patch, HIVE-12205.2.patch,
> HIVE-12205.3.patch
>
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark
> statistics aggregation are done similar but in different code paths. Ideally,
> we should have a unified approach to simply maintenance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11837) comments do not support unicode characters well.
[ https://issues.apache.org/jira/browse/HIVE-11837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengzhaozheng updated HIVE-11837: - Assignee: (was: zengzhaozheng) > comments do not support unicode characters well. > > > Key: HIVE-11837 > URL: https://issues.apache.org/jira/browse/HIVE-11837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.13.1, 1.1.0 > Environment: Hadoop 2.7 > Hive 0.13.1 / Hive 1.1.0 > RHEL 6.4 / SLES 11.3 >Reporter: Rudd Chen >Priority: Minor > Attachments: HIVE-11837.patch > > > the terminal encoding is set to UTF-8, It can display Chinese characters. > then I create a table with a comment in Chinese, both "show create table" and > "desc formatted table" can not display the Chinese characters in the table > comments, meanwhile it can display Chinese characters in column comment.. See > below: > 0: jdbc:hive2://ha-cluster/default> create table tt(id int comment '列中文测试') > comment '表中文测试'; > No rows affected (0.152 seconds) > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> desc formatted tt; > +---+---+-+ > | col_name| data_type > | comment | > +---+---+-+ > | # col_name| data_type > | comment | > | | NULL > | NULL| > | id| int > | 列中文测试 | > | | NULL > | NULL| > | # Detailed Table Information | NULL > | NULL| > | Database: | default > | NULL| > | Owner:| admin > | NULL| > | CreateTime: | Wed Sep 16 11:13:34 CST 2015 > | NULL| > | LastAccessTime: | UNKNOWN > | NULL| > | Protect Mode: | None > | NULL| > | Retention:| 0 > | NULL| > | Location: | hdfs://hacluster/user/hive/warehouse/tt > | NULL| > | Table Type: | MANAGED_TABLE > | NULL| > | Table Parameters: | NULL > | NULL| > | | comment > | \u8868\u4E2D\u6587\u6D4B\u8BD5 | > | | transient_lastDdlTime > | 1442373214 | > | | NULL > | NULL| > | # Storage Information | NULL > | NULL| > | SerDe Library:| > org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe | NULL > | > | InputFormat: | > org.apache.hadoop.hive.ql.io.RCFileInputFormat| NULL > | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.RCFileOutputFormat | NULL > | > | Compressed: | No > | NULL| > | Num Buckets: | -1 > | NULL| > | Bucket Columns: | [] > | NULL| > | Sort Columns: | [] > | NULL| > | Storage Desc Params:
[jira] [Assigned] (HIVE-11837) comments do not support unicode characters well.
[ https://issues.apache.org/jira/browse/HIVE-11837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengzhaozheng reassigned HIVE-11837: Assignee: zengzhaozheng > comments do not support unicode characters well. > > > Key: HIVE-11837 > URL: https://issues.apache.org/jira/browse/HIVE-11837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.13.1, 1.1.0 > Environment: Hadoop 2.7 > Hive 0.13.1 / Hive 1.1.0 > RHEL 6.4 / SLES 11.3 >Reporter: Rudd Chen >Assignee: zengzhaozheng >Priority: Minor > Attachments: HIVE-11837.patch > > > the terminal encoding is set to UTF-8, It can display Chinese characters. > then I create a table with a comment in Chinese, both "show create table" and > "desc formatted table" can not display the Chinese characters in the table > comments, meanwhile it can display Chinese characters in column comment.. See > below: > 0: jdbc:hive2://ha-cluster/default> create table tt(id int comment '列中文测试') > comment '表中文测试'; > No rows affected (0.152 seconds) > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> > 0: jdbc:hive2://ha-cluster/default> desc formatted tt; > +---+---+-+ > | col_name| data_type > | comment | > +---+---+-+ > | # col_name| data_type > | comment | > | | NULL > | NULL| > | id| int > | 列中文测试 | > | | NULL > | NULL| > | # Detailed Table Information | NULL > | NULL| > | Database: | default > | NULL| > | Owner:| admin > | NULL| > | CreateTime: | Wed Sep 16 11:13:34 CST 2015 > | NULL| > | LastAccessTime: | UNKNOWN > | NULL| > | Protect Mode: | None > | NULL| > | Retention:| 0 > | NULL| > | Location: | hdfs://hacluster/user/hive/warehouse/tt > | NULL| > | Table Type: | MANAGED_TABLE > | NULL| > | Table Parameters: | NULL > | NULL| > | | comment > | \u8868\u4E2D\u6587\u6D4B\u8BD5 | > | | transient_lastDdlTime > | 1442373214 | > | | NULL > | NULL| > | # Storage Information | NULL > | NULL| > | SerDe Library:| > org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe | NULL > | > | InputFormat: | > org.apache.hadoop.hive.ql.io.RCFileInputFormat| NULL > | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.RCFileOutputFormat | NULL > | > | Compressed: | No > | NULL| > | Num Buckets: | -1 > | NULL| > | Bucket Columns: | [] > | NULL| > | Sort Columns: | [] > | NULL
[jira] [Commented] (HIVE-13062) Hive metastore test failing
[
https://issues.apache.org/jira/browse/HIVE-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148269#comment-15148269
]
Hive QA commented on HIVE-13062:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12787975/HIVE-13062.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9776 tests executed
*Failed tests:*
{noformat}
TestParseNegative - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6998/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6998/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6998/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12787975 - PreCommit-HIVE-TRUNK-Build
> Hive metastore test failing
> ---
>
> Key: HIVE-13062
> URL: https://issues.apache.org/jira/browse/HIVE-13062
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13062.patch
>
>
> The testutils/metastore/metastore-upgrade-test.sh script takes the list of
> scripts to execute for the DB upgrade from the folder itself
> ({{metastore/scripts/upgrade/DB_NAME}}). Currently it fails as it executes
> upgrade-1.2.0-to-1.3.0 followed by upgrade-1.2.0-to-2.0.0.
> Instead, it should use the content of the {{upgrade.order.DB_NAME}} file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
