[jira] [Assigned] (HIVE-3559) UDF RIGHT(string,position) to HIVE
[
https://issues.apache.org/jira/browse/HIVE-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinaya Varghese reassigned HIVE-3559:
-
Assignee: Vinaya Varghese (was: Meenu K Chandran)
> UDF RIGHT(string,position) to HIVE
> ---
>
> Key: HIVE-3559
> URL: https://issues.apache.org/jira/browse/HIVE-3559
> Project: Hive
> Issue Type: New Feature
> Components: UDF
>Affects Versions: 0.9.0
>Reporter: Vinaya Varghese
>Assignee: Vinaya Varghese
>Priority: Minor
> Attachments: HIVE-3559.1.patch.txt, udf_right.q, udf_right.q.out
>
>
> Introduction
> UDF (User Defined Function) to obtain the rightmost 'n' characters from
> a string in HIVE.
> Relevance
> Current releases of Hive lacks a function which would returns the
> rightmost len characters from the string str, or NULL if any argument is
> NULL. The function LEFT(string,length) would return the rightmost 'length'
> characters from the string 'string' , or NULL if any argument is NULL which
> would be useful while using HiveQL. This would find its use in all the
> technical aspects where the concept of strings are used.
> Functionality :-
> Function Name: RIGHT(string,length)
>
> Returns the rightmost 'length' characters from the string or NULL if any
> argument is NULL.
> Example: hive>SELECT LEFT('https://www.irctc.com',3);
> -> 'com'
> Usage :-
> Case 1: To query a table to find details based on an https request
> Table :-Transaction
> Request_id|date|period_id|url_name
> 0001|01/07/2012|110001|https://www.irctc.com
> 0002|02/07/2012|110001|https://nextstep.tcs.com
> 0003|03/07/2012|110001|https://www.hdfcbank.com
> 0005|01/07/2012|110001|http://www.lmnm.org
> 0006|08/07/2012|110001|http://nextstart.gov
> 0007|10/07/2012|110001|https://netbanking.icicibank.com
> 0012|21/07/2012|110001|http://www.people.nic
> 0026|08/07/2012|110001|http://nextprobs.gov
> 00023|25/07/2012|110001|https://netbanking.canarabank.com
> Query : select * from transaction where RIGHT(url_name,3)='com';
> Result :-
> 0001|01/07/2012|110001|https://www.irctc.com
> 0002|02/07/2012|110001|https://nextstep.tcs.com
> 0003|03/07/2012|110001|https://www.hdfcbank.com
> 0007|10/07/2012|110001|https://netbanking.icicibank.com
> 00023|25/07/2012|110001|https://netbanking.canarabank.com
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3335) Thousand of CLOSE_WAIT socket when we using SymbolicInputFormat
[
https://issues.apache.org/jira/browse/HIVE-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168697#comment-15168697
]
Harsh J commented on HIVE-3335:
---
This was fixed via HIVE-3480
> Thousand of CLOSE_WAIT socket when we using SymbolicInputFormat
> ---
>
> Key: HIVE-3335
> URL: https://issues.apache.org/jira/browse/HIVE-3335
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 0.8.1
> Environment: CentOS 5.8 x64
> CDH3u4
>hadoop-0.20-0.20.2+923.256-1
>
> hadoop-0.20-{namenode,secondarynamenode,jobtracker,tasktracker,datanode}-0.20.2+923.256-1
>hadoop-0.20-conf-pseudo-0.20.2+923.256-1(but same error was
> occurred on not pseudo env)
> apache hive-0.8.1(but same error was occurred on hive 0.9)
>Reporter: Yuki Yoi
> Attachments: HIVE-3335.patch
>
>
> Procedure for reproduction:
> 1. Set up hadoop
> 2. Prepare data file and link.txt:
> data:
> $ hadoop fs -cat /path/to/data/2012-07-01/20120701.csv
> 1, 20120701 00:00:00
> 2, 20120701 00:00:01
> 3, 20120701 01:12:45
> link.txt
> $ cat link.txt
>/path/to/data/2012-07-01//*
> 2. On hive, create table like below:
>CREATE TABLE user_logs(id INT, created_at STRING)
>row format delimited fields terminated by ',' lines terminated by '\n'
>stored as inputformat 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
>outputformat 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
> 3. Put link.txt to /user/hive/warehouse/user_logs
>$ sudo -u hdfs hadoop fs -put link.txt /user/hive/warehouse/user_logs
> 4. Open another session(A session), and watch socket,
>$ netstat -a | grep CLOSE_WAIT
> tcp1 0 localhost:48121 localhost:50010
> CLOSE_WAIT
> tcp1 0 localhost:48124 localhost:50010
> CLOSE_WAIT
>$
> 5. Return to hive session, execute this,
>$ select * from user_logs;
> 6. Return to A session, watch socket again,
>$ netstat -a | grep CLOSE_WAIT
>tcp1 0 localhost:48121 localhost:50010
> CLOSE_WAIT
>tcp1 0 localhost:48124 localhost:50010
> CLOSE_WAIT
>tcp1 0 localhost:48166 localhost:50010
> CLOSE_WAIT
> If you makes any partitions, you'll watch unclosed socket whose count
> equals partitions by once.
> I think that this problem maybe is caused by this point:
> At
> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/SymbolicInputFormat.java,
> line 66. BufferedReader was opened, but it doesn't closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-3335) Thousand of CLOSE_WAIT socket when we using SymbolicInputFormat
[
https://issues.apache.org/jira/browse/HIVE-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HIVE-3335.
---
Resolution: Duplicate
> Thousand of CLOSE_WAIT socket when we using SymbolicInputFormat
> ---
>
> Key: HIVE-3335
> URL: https://issues.apache.org/jira/browse/HIVE-3335
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 0.8.1
> Environment: CentOS 5.8 x64
> CDH3u4
>hadoop-0.20-0.20.2+923.256-1
>
> hadoop-0.20-{namenode,secondarynamenode,jobtracker,tasktracker,datanode}-0.20.2+923.256-1
>hadoop-0.20-conf-pseudo-0.20.2+923.256-1(but same error was
> occurred on not pseudo env)
> apache hive-0.8.1(but same error was occurred on hive 0.9)
>Reporter: Yuki Yoi
> Attachments: HIVE-3335.patch
>
>
> Procedure for reproduction:
> 1. Set up hadoop
> 2. Prepare data file and link.txt:
> data:
> $ hadoop fs -cat /path/to/data/2012-07-01/20120701.csv
> 1, 20120701 00:00:00
> 2, 20120701 00:00:01
> 3, 20120701 01:12:45
> link.txt
> $ cat link.txt
>/path/to/data/2012-07-01//*
> 2. On hive, create table like below:
>CREATE TABLE user_logs(id INT, created_at STRING)
>row format delimited fields terminated by ',' lines terminated by '\n'
>stored as inputformat 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
>outputformat 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
> 3. Put link.txt to /user/hive/warehouse/user_logs
>$ sudo -u hdfs hadoop fs -put link.txt /user/hive/warehouse/user_logs
> 4. Open another session(A session), and watch socket,
>$ netstat -a | grep CLOSE_WAIT
> tcp1 0 localhost:48121 localhost:50010
> CLOSE_WAIT
> tcp1 0 localhost:48124 localhost:50010
> CLOSE_WAIT
>$
> 5. Return to hive session, execute this,
>$ select * from user_logs;
> 6. Return to A session, watch socket again,
>$ netstat -a | grep CLOSE_WAIT
>tcp1 0 localhost:48121 localhost:50010
> CLOSE_WAIT
>tcp1 0 localhost:48124 localhost:50010
> CLOSE_WAIT
>tcp1 0 localhost:48166 localhost:50010
> CLOSE_WAIT
> If you makes any partitions, you'll watch unclosed socket whose count
> equals partitions by once.
> I think that this problem maybe is caused by this point:
> At
> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/SymbolicInputFormat.java,
> line 66. BufferedReader was opened, but it doesn't closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13134) JDBC: JDBC Standalone should not be in the lib dir by default
[
https://issues.apache.org/jira/browse/HIVE-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168744#comment-15168744
]
Hive QA commented on HIVE-13134:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789376/HIVE-13134.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 9814 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_selectDistinctStar
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_noscan_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_multi_insert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_timestamp_funcs
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7096/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7096/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7096/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789376 - PreCommit-HIVE-TRUNK-Build
> JDBC: JDBC Standalone should not be in the lib dir by default
> -
>
> Key: HIVE-13134
> URL: https://issues.apache.org/jira/browse/HIVE-13134
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13134.1.patch
>
>
> JDBC standalone contains many shaded jars, which tends to diverge in version
> when hotfixes are deployed.
> {code}
> $ jar tvf jdbc/target/hive-jdbc-*-standalone.jar | grep slf4j
> 0 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/
> 3366 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarker.class
> 1427 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarkerFactory.class
> 1521 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/FormattingTuple.class
> 4773 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MarkerIgnoringBase.class
> 6699 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MessageFormatter.class
>823 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NamedLoggerBase.class
> 3267 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLogger.class
>584 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLoggerFactory.class
> 1047 Tue Feb 23 17:21:04 PST 2016
> org/slf4j/helpers/SubstituteLoggerFactory.class
>933 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/Util.class
> {code}
> Still need to retain those in the shaded version, but the jar has to out of
> the ./bin/hive classpath to load the service entries in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13134) JDBC: JDBC Standalone should not be in the lib dir by default
[
https://issues.apache.org/jira/browse/HIVE-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168746#comment-15168746
]
Gopal V commented on HIVE-13134:
Failures are unrelated flakiness. The logs contain
{code}
java.lang.RuntimeException: MetaException(message:Unable to read from or write
to hbase Failed 1 action: RetriesExhaustedException: 1 time, )
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:563)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:498)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:478)
at org.apache.hadoop.hive.ql.QTestUtil.init(QTestUtil.java:914)
at org.apache.hadoop.hive.ql.QTestUtil.(QTestUtil.java:475)
at
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.(TestMiniTezCliDriver.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
{code}
> JDBC: JDBC Standalone should not be in the lib dir by default
> -
>
> Key: HIVE-13134
> URL: https://issues.apache.org/jira/browse/HIVE-13134
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13134.1.patch
>
>
> JDBC standalone contains many shaded jars, which tends to diverge in version
> when hotfixes are deployed.
> {code}
> $ jar tvf jdbc/target/hive-jdbc-*-standalone.jar | grep slf4j
> 0 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/
> 3366 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarker.class
> 1427 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarkerFactory.class
> 1521 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/FormattingTuple.class
> 4773 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MarkerIgnoringBase.class
> 6699 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MessageFormatter.class
>823 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NamedLoggerBase.class
> 3267 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLogger.class
>584 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLoggerFactory.class
> 1047 Tue Feb 23 17:21:04 PST 2016
> org/slf4j/helpers/SubstituteLoggerFactory.class
>933 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/Util.class
> {code}
> Still need to retain those in the shaded version, but the jar has to out of
> the ./bin/hive classpath to load the service entries in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13134) JDBC: JDBC Standalone should not be in the lib dir by default
[
https://issues.apache.org/jira/browse/HIVE-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13134:
---
Resolution: Fixed
Fix Version/s: 2.1.0
Release Note: JDBC: JDBC Standalone should not be in the lib dir by
default
Status: Resolved (was: Patch Available)
> JDBC: JDBC Standalone should not be in the lib dir by default
> -
>
> Key: HIVE-13134
> URL: https://issues.apache.org/jira/browse/HIVE-13134
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 2.1.0
>
> Attachments: HIVE-13134.1.patch
>
>
> JDBC standalone contains many shaded jars, which tends to diverge in version
> when hotfixes are deployed.
> {code}
> $ jar tvf jdbc/target/hive-jdbc-*-standalone.jar | grep slf4j
> 0 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/
> 3366 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarker.class
> 1427 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/BasicMarkerFactory.class
> 1521 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/FormattingTuple.class
> 4773 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MarkerIgnoringBase.class
> 6699 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/MessageFormatter.class
>823 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NamedLoggerBase.class
> 3267 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLogger.class
>584 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/NOPLoggerFactory.class
> 1047 Tue Feb 23 17:21:04 PST 2016
> org/slf4j/helpers/SubstituteLoggerFactory.class
>933 Tue Feb 23 17:21:04 PST 2016 org/slf4j/helpers/Util.class
> {code}
> Still need to retain those in the shaded version, but the jar has to out of
> the ./bin/hive classpath to load the service entries in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13135) LLAP: HTTPS Webservices needs trusted keystore configs
[ https://issues.apache.org/jira/browse/HIVE-13135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13135: --- Resolution: Fixed Release Note: LLAP: HTTPS Webservices needs trusted keystore configs Status: Resolved (was: Patch Available) > LLAP: HTTPS Webservices needs trusted keystore configs > -- > > Key: HIVE-13135 > URL: https://issues.apache.org/jira/browse/HIVE-13135 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.1.0 > > Attachments: HIVE-13135.1.patch, HIVE-13135.2.patch > > > ssl-server.xml is not picked up internally by the hive-common HttpServer > impl, unlike the default configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13122) LLAP: simple Model/View separation for UI
[ https://issues.apache.org/jira/browse/HIVE-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13122: --- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > LLAP: simple Model/View separation for UI > - > > Key: HIVE-13122 > URL: https://issues.apache.org/jira/browse/HIVE-13122 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.1.0 > > Attachments: HIVE-13122.1.patch, HIVE-13122.2.patch > > > The current LLAP UI in master uses a fixed loop to both extract data and to > display it in the same loop. > Split this up into a model-view, for modularity. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13068 started by Jesus Camacho Rodriguez. -- > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Attachment: HIVE-13068.patch > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Status: Patch Available (was: In Progress) > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13151) Clean up UGI objects in FileSystem cache for transactions
[
https://issues.apache.org/jira/browse/HIVE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168942#comment-15168942
]
Hive QA commented on HIVE-13151:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790038/HIVE-13151.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 9799 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-auto_sortmerge_join_13.q-tez_self_join.q-alter_merge_2_orc.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadata_only_queries_with_filters
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_dynamic_partition
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_aggregate_without_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_data_types
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_precision
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_nested_udf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_math_funcs
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7097/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7097/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7097/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790038 - PreCommit-HIVE-TRUNK-Build
> Clean up UGI objects in FileSystem cache for transactions
> -
>
> Key: HIVE-13151
> URL: https://issues.apache.org/jira/browse/HIVE-13151
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13151.1.patch
>
>
> One issue with FileSystem.CACHE is that it does not clean itself. The key in
> that cache includes UGI object. When new UGI objects are created and used
> with the FileSystem api, new entries get added to the cache.
> We need to manually clean up those UGI objects once they are no longer in use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168948#comment-15168948
]
Hive QA commented on HIVE-12049:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789882/HIVE-12049.9.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7098/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7098/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7098/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-7098/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
e44198f..5c07894 master -> origin/master
+ git reset --hard HEAD
HEAD is now at e44198f HIVE-12857 : LLAP: modify the decider to allow using
LLAP with whitelisted UDFs (Sergey Shelukhin, reviewed by Gunther Hagleitner)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 5c07894 HIVE-13122: LLAP: simple Model/View separation for UI
(Gopal V)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789882 - PreCommit-HIVE-TRUNK-Build
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.2.patch,
> HIVE-12049.3.patch, HIVE-12049.4.patch, HIVE-12049.5.patch,
> HIVE-12049.6.patch, HIVE-12049.7.patch, HIVE-12049.9.patch
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can re
[jira] [Updated] (HIVE-13034) Add jdeb plugin to build debian
[ https://issues.apache.org/jira/browse/HIVE-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13034: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed the incremental patch. Thanks [~arshadmatin20]! > Add jdeb plugin to build debian > --- > > Key: HIVE-13034 > URL: https://issues.apache.org/jira/browse/HIVE-13034 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Affects Versions: 2.1.0 >Reporter: Arshad Matin >Assignee: Arshad Matin > Fix For: 2.1.0 > > Attachments: HIVE-13034.1.patch, HIVE-13034.patch > > > It would be nice to also generate a debian as a part of build. This can be > done by adding jdeb plugin to dist profile. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13170) HiveAccumuloTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169052#comment-15169052 ] ASF GitHub Bot commented on HIVE-13170: --- GitHub user chutium opened a pull request: https://github.com/apache/hive/pull/66 [HIVE-13170] Implements interface HiveOutputFormat HiveAccumuloTableOutputFormat as OutputFormat in an hive storage handler, should implement HiveOutputFormat interface as well to ensure compatibility ref. https://issues.apache.org/jira/browse/HIVE-13170 You can merge this pull request into a Git repository by running: $ git pull https://github.com/chutium/hive master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/66.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #66 commit 5e658d147072efa9389f44685cf47dc9c5c42ae5 Author: Teng Qiu Date: 2016-02-26T14:00:31Z Implements interface HiveOutputFormat HiveAccumuloTableOutputFormat as OutputFormat in an hive storage handler, should implement HiveOutputFormat interface as well to ensure compatibility ref. https://issues.apache.org/jira/browse/HIVE-13170 > HiveAccumuloTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > --- > > Key: HIVE-13170 > URL: https://issues.apache.org/jira/browse/HIVE-13170 > Project: Hive > Issue Type: Bug > Components: Accumulo Storage Handler >Affects Versions: 1.2.1, 2.0.0 >Reporter: Teng Qiu >Assignee: Josh Elser > > this issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166 > both HiveAccumuloTableOutputFormat and HiveHBaseTableOutputFormat does not > implemented HiveOutputFormat, it may break the compatibility in some other > APIs that are using hive, such as spark's API. > spark expects the OutputFormat called by hive storage handler is some kind of > HiveOutputFormat. which is totally reasonable. > and since they are OutputFormat for hive storage handler, they should not > only extend the 3rd party OutputFormat (AccumuloOutputFormat or > hbase.TableOutputFormat), but also implement HiveOutputFormat interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-13160:
---
Assignee: Aihua Xu (was: Vaibhav Gumashta)
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169065#comment-15169065
]
Aihua Xu commented on HIVE-13160:
-
I will take a look at the issue.
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169084#comment-15169084
]
Aihua Xu commented on HIVE-13129:
-
When HiveServer2 stops, eventually the connection will be released. But if the
server keeps running, we will always have that connection to HMS during the
lifetime while it never gets used after that. And we will have one connection
for each HiveServer2 instance.
That can be considered as leaking I guess.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169089#comment-15169089
]
Aihua Xu commented on HIVE-13149:
-
Those tests don't fail locally. I will reattach the patch to see how it goes.
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some of them like StatsTask, don't need to access
> HMS, currently a new HMS connection will be established for each thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13149:
Status: Open (was: Patch Available)
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some of them like StatsTask, don't need to access
> HMS, currently a new HMS connection will be established for each thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13149:
Attachment: HIVE-13149.1.patch
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some of them like StatsTask, don't need to access
> HMS, currently a new HMS connection will be established for each thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13149:
Attachment: (was: HIVE-13149.1.patch)
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some of them like StatsTask, don't need to access
> HMS, currently a new HMS connection will be established for each thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13149:
Description:
In SessionState class, currently we will always try to get a HMS connection in
{{start(SessionState startSs, boolean isAsync, LogHelper console)}} regardless
of if the connection will be used later or not.
When SessionState is accessed by the tasks in TaskRunner.java, although most of
the tasks other than some like StatsTask, don't need to access HMS. Currently a
new HMS connection will be established for each Task thread. If HiveServer2 is
configured to run in parallel and the query involves many tasks, then the
connections are created but unused.
{noformat}
@Override
public void run() {
runner = Thread.currentThread();
try {
OperationLog.setCurrentOperationLog(operationLog);
SessionState.start(ss);
runSequential();
{noformat}
was:
In SessionState class, currently we will always try to get a HMS connection in
{{start(SessionState startSs, boolean isAsync, LogHelper console)}} regardless
of if the connection will be used later or not.
When SessionState is accessed by the tasks in TaskRunner.java, although most of
the tasks other than some of them like StatsTask, don't need to access HMS,
currently a new HMS connection will be established for each thread. If
HiveServer2 is configured to run in parallel and the query involves many tasks,
then the connections are created but unused.
{noformat}
@Override
public void run() {
runner = Thread.currentThread();
try {
OperationLog.setCurrentOperationLog(operationLog);
SessionState.start(ss);
runSequential();
{noformat}
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13149:
Status: Patch Available (was: Open)
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13153) SessionID is appended to thread name twice
[
https://issues.apache.org/jira/browse/HIVE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169118#comment-15169118
]
Hive QA commented on HIVE-13153:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789891/HIVE-13153.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9828 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7099/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7099/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7099/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789891 - PreCommit-HIVE-TRUNK-Build
> SessionID is appended to thread name twice
> --
>
> Key: HIVE-13153
> URL: https://issues.apache.org/jira/browse/HIVE-13153
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13153.1.patch, HIVE-13153.2.patch
>
>
> HIVE-12249 added sessionId to thread name. In some cases the sessionId could
> be appended twice. Example log line
> {code}
> DEBUG [6432ec22-9f66-4fa5-8770-488a9d3f0b61
> 6432ec22-9f66-4fa5-8770-488a9d3f0b61 main]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13170) HiveAccumuloTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169303#comment-15169303 ] Josh Elser commented on HIVE-13170: --- Took a quick look and left you some comments on the pull-request. Thanks again for submitting the changes. > HiveAccumuloTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > --- > > Key: HIVE-13170 > URL: https://issues.apache.org/jira/browse/HIVE-13170 > Project: Hive > Issue Type: Bug > Components: Accumulo Storage Handler >Affects Versions: 1.2.1, 2.0.0 >Reporter: Teng Qiu >Assignee: Teng Qiu > > this issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166 > both HiveAccumuloTableOutputFormat and HiveHBaseTableOutputFormat does not > implemented HiveOutputFormat, it may break the compatibility in some other > APIs that are using hive, such as spark's API. > spark expects the OutputFormat called by hive storage handler is some kind of > HiveOutputFormat. which is totally reasonable. > and since they are OutputFormat for hive storage handler, they should not > only extend the 3rd party OutputFormat (AccumuloOutputFormat or > hbase.TableOutputFormat), but also implement HiveOutputFormat interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13170) HiveAccumuloTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-13170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HIVE-13170: -- Assignee: Teng Qiu (was: Josh Elser) > HiveAccumuloTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > --- > > Key: HIVE-13170 > URL: https://issues.apache.org/jira/browse/HIVE-13170 > Project: Hive > Issue Type: Bug > Components: Accumulo Storage Handler >Affects Versions: 1.2.1, 2.0.0 >Reporter: Teng Qiu >Assignee: Teng Qiu > > this issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166 > both HiveAccumuloTableOutputFormat and HiveHBaseTableOutputFormat does not > implemented HiveOutputFormat, it may break the compatibility in some other > APIs that are using hive, such as spark's API. > spark expects the OutputFormat called by hive storage handler is some kind of > HiveOutputFormat. which is totally reasonable. > and since they are OutputFormat for hive storage handler, they should not > only extend the 3rd party OutputFormat (AccumuloOutputFormat or > hbase.TableOutputFormat), but also implement HiveOutputFormat interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169312#comment-15169312
]
Chaoyu Tang commented on HIVE-13129:
IMO, it should not be a big problem. This HMS connection can be used to serve
possible HMS requests related to HS2 itself, it can live as long as HS2 does.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12749) Constant propagate returns string values in incorrect format
[
https://issues.apache.org/jira/browse/HIVE-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Vovchenko updated HIVE-12749:
-
Attachment: HIVE-12749.8.patch
> Constant propagate returns string values in incorrect format
>
>
> Key: HIVE-12749
> URL: https://issues.apache.org/jira/browse/HIVE-12749
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.0
>Reporter: Oleksiy Sayankin
>Assignee: Aleksey Vovchenko
> Fix For: 2.0.1
>
> Attachments: HIVE-12749.1.patch, HIVE-12749.2.patch,
> HIVE-12749.3.patch, HIVE-12749.4.patch, HIVE-12749.5.patch,
> HIVE-12749.6.patch, HIVE-12749.7.patch, HIVE-12749.8.patch
>
>
> h2. STEP 1. Create and upload test data
> Execute in command line:
> {noformat}
> nano stest.data
> {noformat}
> Add to file:
> {noformat}
> 000126,000777
> 000126,000778
> 000126,000779
> 000474,000888
> 000468,000889
> 000272,000880
> {noformat}
> {noformat}
> hadoop fs -put stest.data /
> {noformat}
> {noformat}
> hive> create table stest(x STRING, y STRING) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY ',';
> hive> LOAD DATA INPATH '/stest.data' OVERWRITE INTO TABLE stest;
> {noformat}
> h2. STEP 2. Execute test query (with cast for x)
> {noformat}
> select x from stest where cast(x as int) = 126;
> {noformat}
> EXPECTED RESULT:
> {noformat}
> 000126
> 000126
> 000126
> {noformat}
> ACTUAL RESULT:
> {noformat}
> 126
> 126
> 126
> {noformat}
> h2. STEP 3. Execute test query (no cast for x)
> {noformat}
> hive> select x from stest where x = 126;
> {noformat}
> EXPECTED RESULT:
> {noformat}
> 000126
> 000126
> 000126
> {noformat}
> ACTUAL RESULT:
> {noformat}
> 126
> 126
> 126
> {noformat}
> In steps #2, #3 I expected '000126' because the origin type of x is STRING in
> stest table.
> Note, setting hive.optimize.constant.propagation=false fixes the issue.
> {noformat}
> hive> set hive.optimize.constant.propagation=false;
> hive> select x from stest where x = 126;
> OK
> 000126
> 000126
> 000126
> {noformat}
> Related to HIVE-11104, HIVE-8555
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169350#comment-15169350
]
Chaoyu Tang commented on HIVE-13149:
[~aihuaxu] Why not to pass the connection to taskRunner so that it could be
shared by the tasks which need it?
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12935) LLAP: Replace Yarn registry with Zookeeper registry
[
https://issues.apache.org/jira/browse/HIVE-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169375#comment-15169375
]
Hive QA commented on HIVE-12935:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790078/HIVE-12935.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9829 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_partitioned
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7100/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7100/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7100/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790078 - PreCommit-HIVE-TRUNK-Build
> LLAP: Replace Yarn registry with Zookeeper registry
> ---
>
> Key: HIVE-12935
> URL: https://issues.apache.org/jira/browse/HIVE-12935
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: 12935.1.patch, HIVE-12935.2.patch, HIVE-12935.3.patch,
> HIVE-12935.4.patch, HIVE-12935.5.patch, HIVE-12935.6.patch, HIVE-12935.7.patch
>
>
> Existing YARN registry service for cluster membership has to depend on
> refresh intervals to get the list of instances/daemons that are running in
> the cluster. Better approach would be replace it with zookeeper based
> registry service so that custom listeners can be added to update healthiness
> of daemons in the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169470#comment-15169470
]
Aihua Xu commented on HIVE-13129:
-
If it's used, it's OK. But right now it's not used after that.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169492#comment-15169492
]
Aihua Xu commented on HIVE-13149:
-
That's is about performance. We actually can share one HMS connection in
extreme in one HS2 but sharing the connection will cause the one HMS execution
to wait for the other long HMS execution.
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12761) Add stack trace servlet to HS2 web ui
[ https://issues.apache.org/jira/browse/HIVE-12761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-12761: - Issue Type: Sub-task (was: Improvement) Parent: HIVE-12338 > Add stack trace servlet to HS2 web ui > - > > Key: HIVE-12761 > URL: https://issues.apache.org/jira/browse/HIVE-12761 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita >Priority: Minor > Fix For: 2.0.0, 2.1.0 > > Attachments: HIVE-12761.1.patch, HIVE-12761.2.patch, > HIVE-12761.3.patch, ScreenShot1.png, ScreenShot2.png > > > To confirm the state of HS2, I add the servlet which prints stack trace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-12338) Add webui to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho resolved HIVE-12338. -- Resolution: Fixed I am not too sure the correct precedent, but I think we can mark this JIRA as resolved for 2.0 as the JIRA's patch is in 2.0 .. although some sub-tasks may be beyond 2.0. > Add webui to HiveServer2 > > > Key: HIVE-12338 > URL: https://issues.apache.org/jira/browse/HIVE-12338 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: HIVE-12338.1.patch, HIVE-12338.2.patch, > HIVE-12338.3.patch, HIVE-12338.4.patch, hs2-conf.png, hs2-logs.png, > hs2-metrics.png, hs2-webui.png > > > A web ui for HiveServer2 can show some useful information such as: > > 1. Sessions, > 2. Queries that are executing on the HS2, their states, starting time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13002) metastore call timing is not threadsafe
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169528#comment-15169528
]
Sergey Shelukhin commented on HIVE-13002:
-
[~ashutoshc] ping?
> metastore call timing is not threadsafe
> ---
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13108) Operators: SORT BY randomness is not safe with network partitions
[ https://issues.apache.org/jira/browse/HIVE-13108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169575#comment-15169575 ] Sergey Shelukhin commented on HIVE-13108: - +1 > Operators: SORT BY randomness is not safe with network partitions > - > > Key: HIVE-13108 > URL: https://issues.apache.org/jira/browse/HIVE-13108 > Project: Hive > Issue Type: Bug > Components: Spark, Tez >Affects Versions: 1.3.0, 1.2.1, 2.0.0, 2.0.1 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13108.1.patch > > > SORT BY relies on a transient Random object, which is initialized once per > deserialize operation. > This results in complications during a network partition and when Tez/Spark > reuses a cached plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13096) Cost to choose side table in MapJoin conversion based on cumulative cardinality
[
https://issues.apache.org/jira/browse/HIVE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169587#comment-15169587
]
Hive QA commented on HIVE-13096:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789935/HIVE-13096.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9828 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7101/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7101/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7101/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789935 - PreCommit-HIVE-TRUNK-Build
> Cost to choose side table in MapJoin conversion based on cumulative
> cardinality
> ---
>
> Key: HIVE-13096
> URL: https://issues.apache.org/jira/browse/HIVE-13096
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13096.01.patch, HIVE-13096.02.patch,
> HIVE-13096.03.patch, HIVE-13096.patch
>
>
> HIVE-11954 changed the logic to choose the side table in the MapJoin
> conversion algorithm. Initial heuristic for the cost was based on number of
> heavyweight operators.
> This extends that work so the heuristic is based on accumulate cardinality.
> In the future, we should choose the side based on total latency for the input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13169) HiveServer2: Support delegation token based connection when using http transport
[ https://issues.apache.org/jira/browse/HIVE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13169: Attachment: HIVE-13169.1.patch > HiveServer2: Support delegation token based connection when using http > transport > > > Key: HIVE-13169 > URL: https://issues.apache.org/jira/browse/HIVE-13169 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-13169.1.patch > > > HIVE-5155 introduced support for delegation token based connection. However, > it was intended for tcp transport mode. We need to have similar mechanisms > for http transport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13169) HiveServer2: Support delegation token based connection when using http transport
[ https://issues.apache.org/jira/browse/HIVE-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13169: Status: Patch Available (was: Open) > HiveServer2: Support delegation token based connection when using http > transport > > > Key: HIVE-13169 > URL: https://issues.apache.org/jira/browse/HIVE-13169 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Affects Versions: 2.0.0, 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-13169.1.patch > > > HIVE-5155 introduced support for delegation token based connection. However, > it was intended for tcp transport mode. We need to have similar mechanisms > for http transport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13002) metastore call timing is not threadsafe
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169642#comment-15169642
]
Deepesh Khandelwal commented on HIVE-13002:
---
[~sershe] any reason why you replaced all the Map<...> with
ConcurrentHashMap<...>? I would think that just changing the implementation at
the initialization time would be enough and better.
> metastore call timing is not threadsafe
> ---
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13160:
Attachment: HIVE-13160.1.patch
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Attachments: HIVE-13160.1.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169649#comment-15169649
]
Aihua Xu commented on HIVE-13160:
-
Attached the patch-1: we will throw the exception when function registering
fails and the caller can decide to rethrow or consume it. The HiveServer will
be notified and attempt to restart.
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Attachments: HIVE-13160.1.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIR
[jira] [Updated] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13160:
Status: Patch Available (was: Open)
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Attachments: HIVE-13160.1.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[ https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11675: Attachment: HIVE-11675.07.patch Addressed RB feedback. Also refactored all the cache classes out of OrcInputFormat to avoid semanticanalyzerization ;) > make use of file footer PPD API in ETL strategy or separate strategy > > > Key: HIVE-11675 > URL: https://issues.apache.org/jira/browse/HIVE-11675 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch, > HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch, > HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.patch > > > Need to take a look at the best flow. It won't be much different if we do > filtering metastore call for each partition. So perhaps we'd need the custom > sync point/batching after all. > Or we can make it opportunistic and not fetch any footers unless it can be > pushed down to metastore or fetched from local cache, that way the only slow > threaded op is directory listings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[ https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169715#comment-15169715 ] Sergey Shelukhin commented on HIVE-11675: - [~prasanth_j] can you review? The new files are mostly moved/renamed code, with some small changes. Every time this patch sits for a while it gets a lot of conflicts. > make use of file footer PPD API in ETL strategy or separate strategy > > > Key: HIVE-11675 > URL: https://issues.apache.org/jira/browse/HIVE-11675 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch, > HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch, > HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.patch > > > Need to take a look at the best flow. It won't be much different if we do > filtering metastore call for each partition. So perhaps we'd need the custom > sync point/batching after all. > Or we can make it opportunistic and not fetch any footers unless it can be > pushed down to metastore or fetched from local cache, that way the only slow > threaded op is directory listings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HIVE-11644) make sure Hive config is propagated to AM-side split generation
[ https://issues.apache.org/jira/browse/HIVE-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin closed HIVE-11644. --- > make sure Hive config is propagated to AM-side split generation > --- > > Key: HIVE-11644 > URL: https://issues.apache.org/jira/browse/HIVE-11644 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > When trying to access file cache, Tez AM creates local metastore despite the > uris being specified and visible on the CLI. > Somewhere in the zoo of hive configs, tez configs, job configs, contexts, > descriptors, user payloads etc. it gets lost. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-1955) Support non-constant expressions for array indexes.
[ https://issues.apache.org/jira/browse/HIVE-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169718#comment-15169718 ] Shawn Lavelle commented on HIVE-1955: - Was this ever addressed? > Support non-constant expressions for array indexes. > --- > > Key: HIVE-1955 > URL: https://issues.apache.org/jira/browse/HIVE-1955 > Project: Hive > Issue Type: Improvement >Reporter: Adam Kramer >Assignee: Navis > Fix For: 0.14.0 > > > FAILED: Error in semantic analysis: line 4:8 Non Constant Expressions for > Array Indexes not Supported dut > ...just wrote my own UDF to do this, and it is trivial. We should support > this natively. > Let foo have these rows: > arr i > [1,2,3] 1 > [3,4,5] 2 > [5,4,3] 2 > [0,0,1] 0 > Then, > SELECT arr[i] FROM foo > should return: > 2 > 5 > 3 > 1 > Similarly, for the same table, > SELECT 3 IN arr FROM foo > should return: > true > true > true > false > ...these use cases are needless limitations of functionality. We shouldn't > need UDFs to accomplish these goals. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169726#comment-15169726
]
Aihua Xu commented on HIVE-13160:
-
[~sershe] Can you take a look at the change since you added the logic? I'm
wondering if you are expecting other places to use registerAllFunctionsOnce().
Seems to me we should only call it once when the server starts.
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Attachments: HIVE-13160.1.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message w
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169738#comment-15169738
]
Sergey Shelukhin commented on HIVE-13160:
-
I only added the safety logic, not the original logic ;)
The state still needs to be changed, otherwise as far as I see that will just
leave it "in progress".
Also, would someone register them later, even if the state is updated back to
not-started?
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Attachments: HIVE-13160.1.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better
[jira] [Commented] (HIVE-12935) LLAP: Replace Yarn registry with Zookeeper registry
[
https://issues.apache.org/jira/browse/HIVE-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169817#comment-15169817
]
Prasanth Jayachandran commented on HIVE-12935:
--
Tested this patch on 1TB scale q55-random.sql queries with chaosmonkey interval
of 120s on 6 instances. However I did not see any logs related to chaosmonkey
terminating an instance. The entire queryset succeeded though without any
issues. I also ran the same test with slider flexing up/down the number of
containers and was able to see the notifications. All queries succeeded while
flexing up and down (with few attempt failures/kills which are expected on
flexing down).
{code}
2016-02-26 16:03:23,685 [INFO] [StateChangeNotificationHandler]
|impl.LlapZookeeperRegistryImpl$InstanceStateChangeListener|: Removed zknode
/prasanth/llap0/workers/worker-000106 from llap namespace. Notifying state
change listener.
2016-02-26 16:03:23,685 [INFO] [StateChangeNotificationHandler]
|tezplugins.LlapTaskSchedulerService$NodeStateChangeListener|: Removed node
with identity: 8b46e9f6-eefd-40d7-b17c-5b3016039934
{code}
> LLAP: Replace Yarn registry with Zookeeper registry
> ---
>
> Key: HIVE-12935
> URL: https://issues.apache.org/jira/browse/HIVE-12935
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: 12935.1.patch, HIVE-12935.2.patch, HIVE-12935.3.patch,
> HIVE-12935.4.patch, HIVE-12935.5.patch, HIVE-12935.6.patch, HIVE-12935.7.patch
>
>
> Existing YARN registry service for cluster membership has to depend on
> refresh intervals to get the list of instances/daemons that are running in
> the cluster. Better approach would be replace it with zookeeper based
> registry service so that custom listeners can be added to update healthiness
> of daemons in the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13160:
Status: Open (was: Patch Available)
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Attachments: HIVE-13160.1.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12935) LLAP: Replace Yarn registry with Zookeeper registry
[ https://issues.apache.org/jira/browse/HIVE-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12935: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed patch to master. Thanks [~gopalv] and [~sseth] for the reviews! > LLAP: Replace Yarn registry with Zookeeper registry > --- > > Key: HIVE-12935 > URL: https://issues.apache.org/jira/browse/HIVE-12935 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.1.0 > > Attachments: 12935.1.patch, HIVE-12935.2.patch, HIVE-12935.3.patch, > HIVE-12935.4.patch, HIVE-12935.5.patch, HIVE-12935.6.patch, HIVE-12935.7.patch > > > Existing YARN registry service for cluster membership has to depend on > refresh intervals to get the list of instances/daemons that are running in > the cluster. Better approach would be replace it with zookeeper based > registry service so that custom listeners can be added to update healthiness > of daemons in the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13002) metastore call timing is not threadsafe
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169856#comment-15169856
]
Ashutosh Chauhan commented on HIVE-13002:
-
I dont think we really have a choice to ignore root cause. Its better to fix
caching of ThreadLocal Hive variable. Performance (of access via Hive.get())
can't get in a way of correctness.
> metastore call timing is not threadsafe
> ---
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13013) Further Improve concurrency in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13013: -- Attachment: HIVE-13013.4.patch TestTxnHandlerNegative.testBadConnection is genuine. patch 4 addresses it > Further Improve concurrency in TxnHandler > - > > Key: HIVE-13013 > URL: https://issues.apache.org/jira/browse/HIVE-13013 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13013.2.patch, HIVE-13013.3.patch, > HIVE-13013.4.patch, HIVE-13013.patch > > > There are still a few operations in TxnHandler that run at Serializable > isolation. > Most or all of them can be dropped to READ_COMMITTED now that we have SELECT > ... FOR UPDATE support. This will reduce number of deadlocks in the DBs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Attachment: HIVE-13063.patch
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Status: Open (was: Patch Available)
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Attachment: (was: HIVE-13063.patch)
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: Screen Shot 2016-02-17 at 7.20.57 PM.png, Screen Shot
> 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Status: Patch Available (was: Open)
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169902#comment-15169902
]
Chaoyu Tang commented on HIVE-13149:
It might not be proper to share the HMS connection opened in HS2 main process,
but I think the session one could be shared which can avoid opening connection
in the spawned task thread, is not it? In addition, when these threads finish
the tasks and die, should not they be collected?
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13160:
Attachment: HIVE-13160.2.patch
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13160:
Status: Patch Available (was: Open)
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169937#comment-15169937
]
Aihua Xu commented on HIVE-13160:
-
Updated the patch-2 so if it fails the state will be marked as NO so HiveServer
will attempt to register again and start.
Thanks for the info. :) If start is updated to NO somehow, it will be
reregistered.
I was a little concerned about the comment you put and kind of scared of making
the change. :). We don't care about which metastore we are talking to since
it's retrieving the data from the database, right?
{noformat}
// Note that while this is an improvement over static initialization, it is
still not,
// technically, valid, cause nothing prevents us from connecting to several
metastores in
// the same process. This will still only get the functions from the first
metastore.
{noformat}
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then no
[jira] [Commented] (HIVE-13131) TezWork queryName can be null after HIVE-12523
[
https://issues.apache.org/jira/browse/HIVE-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169943#comment-15169943
]
Hive QA commented on HIVE-13131:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789972/HIVE-13131.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9813 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-unionDistinct_1.q-insert_update_delete.q-selectDistinctStar.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7102/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7102/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7102/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789972 - PreCommit-HIVE-TRUNK-Build
> TezWork queryName can be null after HIVE-12523
> --
>
> Key: HIVE-13131
> URL: https://issues.apache.org/jira/browse/HIVE-13131
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-13131.1.patch, HIVE-13131.2.patch
>
>
> Looks like after HIVE-12523, the queryName field can be null, either if the
> conf passed in is null, or if the conf does not contain the necessary
> settings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13151) Clean up UGI objects in FileSystem cache for transactions
[ https://issues.apache.org/jira/browse/HIVE-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13151: - Attachment: HIVE-13151.2.patch > Clean up UGI objects in FileSystem cache for transactions > - > > Key: HIVE-13151 > URL: https://issues.apache.org/jira/browse/HIVE-13151 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13151.1.patch, HIVE-13151.2.patch > > > One issue with FileSystem.CACHE is that it does not clean itself. The key in > that cache includes UGI object. When new UGI objects are created and used > with the FileSystem api, new entries get added to the cache. > We need to manually clean up those UGI objects once they are no longer in use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169974#comment-15169974
]
Aihua Xu commented on HIVE-13149:
-
You mean HMS connections? Yeah, they will be closed.
Session one is mainly used for query compilation which may hold the connection
for some time. Feel better not to share it. With the fix, actually we don't
need to open many connections.
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13146) OrcFile table property values are case sensitive
[ https://issues.apache.org/jira/browse/HIVE-13146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-13146: Attachment: HIVE-13146.2.patch Attach patch 2 to fix the issue in a general way. Thanks [~aihuaxu] for your suggestion. > OrcFile table property values are case sensitive > > > Key: HIVE-13146 > URL: https://issues.apache.org/jira/browse/HIVE-13146 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.2.1 >Reporter: Andrew Sears >Assignee: Yongzhi Chen >Priority: Minor > Attachments: HIVE-13146.1.patch, HIVE-13146.2.patch > > > In Hive v1.2.1.2.3, with Tez , create an external table with compression > SNAPPY value marked as lower case. Table is created successfully. Insert > data into table fails with no enum constant error. > CREATE EXTERNAL TABLE mydb.mytable > (id int) > PARTITIONED BY (business_date date) > STORED AS ORC > LOCATION > '/data/mydb/mytable' > TBLPROPERTIES ( > 'orc.compress'='snappy'); > set hive.exec.dynamic.partition=true; > set hive.exec.dynamic.partition.mode=nonstrict; > INSERT OVERWRITE mydb.mytable PARTITION (business_date) > SELECT * from mydb.sourcetable; > Caused by: java.lang.IllegalArgumentException: No enum constant > org.apache.hadoop.hive.ql.io.orc.CompressionKind.snappy > at java.lang.Enum.valueOf(Enum.java:238) > at > org.apache.hadoop.hive.ql.io.orc.CompressionKind.valueOf(CompressionKind.java:25) > Constant SNAPPY needs to be uppercase in definition to fix. Case should be > agnostic or throw error on creation of table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170049#comment-15170049
]
Yongzhi Chen commented on HIVE-13160:
-
The second patch looks good.
+1
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13131) TezWork queryName can be null after HIVE-12523
[ https://issues.apache.org/jira/browse/HIVE-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13131: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master > TezWork queryName can be null after HIVE-12523 > -- > > Key: HIVE-13131 > URL: https://issues.apache.org/jira/browse/HIVE-13131 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 2.1.0 > > Attachments: HIVE-13131.1.patch, HIVE-13131.2.patch > > > Looks like after HIVE-12523, the queryName field can be null, either if the > conf passed in is null, or if the conf does not contain the necessary > settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13172) Merge master into llap branch
[ https://issues.apache.org/jira/browse/HIVE-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere resolved HIVE-13172. --- Resolution: Fixed Fix Version/s: llap merge done. > Merge master into llap branch > - > > Key: HIVE-13172 > URL: https://issues.apache.org/jira/browse/HIVE-13172 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: llap > > > This is to get HIVE-13131 into the llap branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13174) Remove Vectorizer noise in logs
[
https://issues.apache.org/jira/browse/HIVE-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-13174:
-
Attachment: HIVE-13174.patch
> Remove Vectorizer noise in logs
> ---
>
> Key: HIVE-13174
> URL: https://issues.apache.org/jira/browse/HIVE-13174
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13174.patch
>
>
> If you have a table with a bin column you're hs2/client logs are full of the
> stack traces below. These should either be made debug or we just log the
> message not the trace.
> {code}
> 2015-10-12 12:34:23,922 INFO [main]: physical.Vectorizer
> (Vectorizer.java:validateExprNodeDesc(1249)) - Failed to vectorize
> org.apache.hadoop.hive.ql.metadata.HiveException: No vector argument type for
> type name binary
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getConstantVectorExpression(VectorizationContext.java:872)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:443)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateExprNodeDesc(Vectorizer.java:1243)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateExprNodeDesc(Vectorizer.java:1234)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateSelectOperator(Vectorizer.java:1100)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateMapWorkOperator(Vectorizer.java:911)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$MapWorkValidationNodeProcessor.process(Vectorizer.java:581)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateMapWork(Vectorizer.java:412)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:355)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:330)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:890)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:469)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10188)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:211)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.refl
[jira] [Commented] (HIVE-13146) OrcFile table property values are case sensitive
[
https://issues.apache.org/jira/browse/HIVE-13146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170134#comment-15170134
]
Hive QA commented on HIVE-13146:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789977/HIVE-13146.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9828 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7103/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7103/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7103/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789977 - PreCommit-HIVE-TRUNK-Build
> OrcFile table property values are case sensitive
>
>
> Key: HIVE-13146
> URL: https://issues.apache.org/jira/browse/HIVE-13146
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 1.2.1
>Reporter: Andrew Sears
>Assignee: Yongzhi Chen
>Priority: Minor
> Attachments: HIVE-13146.1.patch, HIVE-13146.2.patch
>
>
> In Hive v1.2.1.2.3, with Tez , create an external table with compression
> SNAPPY value marked as lower case. Table is created successfully. Insert
> data into table fails with no enum constant error.
> CREATE EXTERNAL TABLE mydb.mytable
> (id int)
> PARTITIONED BY (business_date date)
> STORED AS ORC
> LOCATION
> '/data/mydb/mytable'
> TBLPROPERTIES (
> 'orc.compress'='snappy');
> set hive.exec.dynamic.partition=true;
> set hive.exec.dynamic.partition.mode=nonstrict;
> INSERT OVERWRITE mydb.mytable PARTITION (business_date)
> SELECT * from mydb.sourcetable;
> Caused by: java.lang.IllegalArgumentException: No enum constant
> org.apache.hadoop.hive.ql.io.orc.CompressionKind.snappy
> at java.lang.Enum.valueOf(Enum.java:238)
> at
> org.apache.hadoop.hive.ql.io.orc.CompressionKind.valueOf(CompressionKind.java:25)
> Constant SNAPPY needs to be uppercase in definition to fix. Case should be
> agnostic or throw error on creation of table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-8323) Ensure "transactional" tbl property can only be set on tables using AcidOutputFormat
[ https://issues.apache.org/jira/browse/HIVE-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng resolved HIVE-8323. - Resolution: Fixed This was covered by HIVE-12064 > Ensure "transactional" tbl property can only be set on tables using > AcidOutputFormat > > > Key: HIVE-8323 > URL: https://issues.apache.org/jira/browse/HIVE-8323 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.14.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13174) Remove Vectorizer noise in logs
[
https://issues.apache.org/jira/browse/HIVE-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-13174:
-
Status: Patch Available (was: Open)
> Remove Vectorizer noise in logs
> ---
>
> Key: HIVE-13174
> URL: https://issues.apache.org/jira/browse/HIVE-13174
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13174.patch
>
>
> If you have a table with a bin column you're hs2/client logs are full of the
> stack traces below. These should either be made debug or we just log the
> message not the trace.
> {code}
> 2015-10-12 12:34:23,922 INFO [main]: physical.Vectorizer
> (Vectorizer.java:validateExprNodeDesc(1249)) - Failed to vectorize
> org.apache.hadoop.hive.ql.metadata.HiveException: No vector argument type for
> type name binary
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getConstantVectorExpression(VectorizationContext.java:872)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:443)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateExprNodeDesc(Vectorizer.java:1243)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateExprNodeDesc(Vectorizer.java:1234)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateSelectOperator(Vectorizer.java:1100)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateMapWorkOperator(Vectorizer.java:911)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$MapWorkValidationNodeProcessor.process(Vectorizer.java:581)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateMapWork(Vectorizer.java:412)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:355)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:330)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:890)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:469)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10188)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:211)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
[jira] [Commented] (HIVE-12637) make retryable SQLExceptions in TxnHandler configurable
[
https://issues.apache.org/jira/browse/HIVE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170160#comment-15170160
]
Wei Zheng commented on HIVE-12637:
--
[~ekoifman] Can you give a concrete example for this?
> make retryable SQLExceptions in TxnHandler configurable
> ---
>
> Key: HIVE-12637
> URL: https://issues.apache.org/jira/browse/HIVE-12637
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>
> same for CompactionTxnHandler
> would be convenient if the user could specify some RegEx (perhaps by db type)
> which will tell TxnHandler.checkRetryable() that this is should be retried.
> The regex should probably apply to String produced by
> {noformat}
> private static String getMessage(SQLException ex) {
> return ex.getMessage() + "(SQLState=" + ex.getSQLState() + ",ErrorCode="
> + ex.getErrorCode() + ")";
> }
> {noformat}
> This make it flexible.
> See if we need to add Db type (and possibly version) of the DB being used.
> With 5 different DBs supported this gives control end users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13040) Handle empty bucket creations more efficiently
[ https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170244#comment-15170244 ] Prasanth Jayachandran commented on HIVE-13040: -- Minor comment in RB. Otherwise looks good to me, +1 > Handle empty bucket creations more efficiently > --- > > Key: HIVE-13040 > URL: https://issues.apache.org/jira/browse/HIVE-13040 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch, > HIVE-13040.4.patch, HIVE-13040.5.patch, HIVE-13040.6.patch, > HIVE-13040.7.patch, HIVE-13040.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13002) metastore call timing is not threadsafe
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13002:
Attachment: HIVE-13002.02.patch
Ok, here's a better patch. This one removes some fields and adds comments to
the ones that appear safe.
> metastore call timing is not threadsafe
> ---
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13069) Enable cartesian product merging
[
https://issues.apache.org/jira/browse/HIVE-13069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170304#comment-15170304
]
Hive QA commented on HIVE-13069:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790006/HIVE-13069.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 9813 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query28
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query65
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query88
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_outer_join5
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7104/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7104/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7104/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790006 - PreCommit-HIVE-TRUNK-Build
> Enable cartesian product merging
>
>
> Key: HIVE-13069
> URL: https://issues.apache.org/jira/browse/HIVE-13069
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13069.patch
>
>
> Currently we can merge 2-way joins into n-way joins when the joins are
> executed over the same column.
> In turn, CBO might produce plans containing cartesian products if the join
> columns are constant values; after HIVE-12543 went in, this is rather common,
> as those constant columns are correctly pruned. However, currently we do not
> merge a cartesian product with two inputs into a cartesian product with
> multiple inputs, which could result in performance loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13002) metastore call timing is not threadsafe
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170303#comment-15170303
]
Sergey Shelukhin edited comment on HIVE-13002 at 2/27/16 2:55 AM:
--
Ok, here's a better patch. This one removes some fields and adds comments to
the ones that appear safe. cc [~ashutoshc]
was (Author: sershe):
Ok, here's a better patch. This one removes some fields and adds comments to
the ones that appear safe.
> metastore call timing is not threadsafe
> ---
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13002:
Summary: Hive object is not thread safe, is shared via a threadlocal and
thus should not be passed around too much (was: metastore call timing is not
threadsafe)
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13002:
Summary: Hive object is not thread safe, is shared via a threadlocal and
thus should not be passed around too much - part 1 (was: Hive object is not
thread safe, is shared via a threadlocal and thus should not be passed around
too much )
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much - part 1
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170306#comment-15170306
]
Sergey Shelukhin commented on HIVE-13002:
-
https://reviews.apache.org/r/44128/
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much - part 1
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13002) Hive object is not thread safe, is shared via a threadlocal and thus should not be passed around too much - part 1
[
https://issues.apache.org/jira/browse/HIVE-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170308#comment-15170308
]
Sergey Shelukhin commented on HIVE-13002:
-
cc also [~thejas], lots of changes in auth plugins, I hope I didn't miss any
other reflection with ctors
> Hive object is not thread safe, is shared via a threadlocal and thus should
> not be passed around too much - part 1
> --
>
> Key: HIVE-13002
> URL: https://issues.apache.org/jira/browse/HIVE-13002
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13002.01.patch, HIVE-13002.02.patch,
> HIVE-13002.patch
>
>
> Discovered in some q test run:
> {noformat}
> TestCliDriver.testCliDriver_insert_values_orig_table:123->runTest:199
> Unexpected exception java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$EntryIterator.next(HashMap.java:966)
> at java.util.HashMap$EntryIterator.next(HashMap.java:964)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dumpAndClearMetaCallTiming(Hive.java:3412)
> at
> org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:574)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1722)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1342)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Attachment: (was: HIVE-13063.patch)
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: Screen Shot 2016-02-17 at 7.20.57 PM.png, Screen Shot
> 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Attachment: HIVE-13063.patch
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12994) Implement support for NULLS FIRST/NULLS LAST
[
https://issues.apache.org/jira/browse/HIVE-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170386#comment-15170386
]
Hive QA commented on HIVE-12994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790020/HIVE-12994.11.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9816 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7105/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7105/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7105/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790020 - PreCommit-HIVE-TRUNK-Build
> Implement support for NULLS FIRST/NULLS LAST
>
>
> Key: HIVE-12994
> URL: https://issues.apache.org/jira/browse/HIVE-12994
> Project: Hive
> Issue Type: New Feature
> Components: CBO, Parser, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12994.01.patch, HIVE-12994.02.patch,
> HIVE-12994.03.patch, HIVE-12994.04.patch, HIVE-12994.05.patch,
> HIVE-12994.06.patch, HIVE-12994.06.patch, HIVE-12994.07.patch,
> HIVE-12994.08.patch, HIVE-12994.09.patch, HIVE-12994.10.patch,
> HIVE-12994.11.patch, HIVE-12994.patch
>
>
> From SQL:2003, the NULLS FIRST and NULLS LAST options can be used to
> determine whether nulls appear before or after non-null data values when the
> ORDER BY clause is used.
> SQL standard does not specify the behavior by default. Currently in Hive,
> null values sort as if lower than any non-null value; that is, NULLS FIRST is
> the default for ASC order, and NULLS LAST for DESC order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13173) LLAP: Add end-to-end test for LlapInputFormat
[ https://issues.apache.org/jira/browse/HIVE-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170422#comment-15170422 ] Lefty Leverenz commented on HIVE-13173: --- [~jdere], you committed this so the status should be updated. (See commit d8a9531a7738ad0137e287c4664a45b934e97c1e.) > LLAP: Add end-to-end test for LlapInputFormat > - > > Key: HIVE-13173 > URL: https://issues.apache.org/jira/browse/HIVE-13173 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Jason Dere >Assignee: Jason Dere > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9499) hive.limit.query.max.table.partition makes queries fail on non-partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-9499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170431#comment-15170431
]
Hive QA commented on HIVE-9499:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12697379/HIVE-9499.3.patch.txt
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9828 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7106/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7106/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7106/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12697379 - PreCommit-HIVE-TRUNK-Build
> hive.limit.query.max.table.partition makes queries fail on non-partitioned
> tables
> -
>
> Key: HIVE-9499
> URL: https://issues.apache.org/jira/browse/HIVE-9499
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Alexander Kasper
>Assignee: Navis
> Attachments: HIVE-9499.1.patch.txt, HIVE-9499.2.patch.txt,
> HIVE-9499.3.patch.txt
>
>
> If you use hive.limit.query.max.table.partition to limit the amount of
> partitions that can be queried it makes queries on non-partitioned tables
> fail.
> Example:
> {noformat}
> CREATE TABLE tmp(test INT);
> SELECT COUNT(*) FROM TMP; -- works fine
> SET hive.limit.query.max.table.partition=20;
> SELECT COUNT(*) FROM TMP; -- generates NPE (FAILED: NullPointerException null)
> SET hive.limit.query.max.table.partition=-1;
> SELECT COUNT(*) FROM TMP; -- works fine again
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
