[jira] [Commented] (HIVE-20607) TxnHandler should use PreparedStatement to execute direct SQL queries.
[ https://issues.apache.org/jira/browse/HIVE-20607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661476#comment-16661476 ] Daniel Dai commented on HIVE-20607: --- [~sankarh], can you commit it to branch-3 as well? > TxnHandler should use PreparedStatement to execute direct SQL queries. > -- > > Key: HIVE-20607 > URL: https://issues.apache.org/jira/browse/HIVE-20607 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore, Transactions >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20607.01.patch > > > TxnHandler uses direct SQL queries to operate on Txn related databases/tables > in Hive metastore RDBMS. > Most of the methods are direct calls from Metastore api which should be > directly append input string arguments to the SQL string. > Need to use parameterised PreparedStatement object to set these arguments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20420) Provide a fallback authorizer when no other authorizer is in use
[ https://issues.apache.org/jira/browse/HIVE-20420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20420: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.3.4 3.1.1 4.0.0 Status: Resolved (was: Patch Available) Patch pushed to master/branch-3/branch-2/branch-3.1/branch-2.3. Thanks Laszlo, Thejas for review! > Provide a fallback authorizer when no other authorizer is in use > > > Key: HIVE-20420 > URL: https://issues.apache.org/jira/browse/HIVE-20420 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0, 3.1.1, 2.3.4 > > Attachments: HIVE-20420.1.patch, HIVE-20420.2.patch, > HIVE-20420.3.patch, HIVE-20420.4.patch, HIVE-20420.5.patch, > HIVE-20420.6.patch, HIVE-20420.7.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20420) Provide a fallback authorizer when no other authorizer is in use
[ https://issues.apache.org/jira/browse/HIVE-20420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661451#comment-16661451 ] Thejas M Nair commented on HIVE-20420: -- +1 > Provide a fallback authorizer when no other authorizer is in use > > > Key: HIVE-20420 > URL: https://issues.apache.org/jira/browse/HIVE-20420 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20420.1.patch, HIVE-20420.2.patch, > HIVE-20420.3.patch, HIVE-20420.4.patch, HIVE-20420.5.patch, > HIVE-20420.6.patch, HIVE-20420.7.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20793: --- > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20772) record per-task CPU counters in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20772: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > record per-task CPU counters in LLAP > > > Key: HIVE-20772 > URL: https://issues.apache.org/jira/browse/HIVE-20772 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20772.01.patch, HIVE-20772.02.patch, > HIVE-20772.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661379#comment-16661379
]
Sergey Shelukhin commented on HIVE-20772:
-
{noformat}
+ java -cp
'/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-3.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
org.apache.hive.ptest.api.client.PTestClient --command testStart --outputDir
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
--password '[***]' --testHandle PreCommit-HIVE-Build-14608 --endpoint
http://104.198.109.242:8080/hive-ptest-3.0 --logsEndpoint
http://104.198.109.242/logs/ --profile master-mr2 --patch
https://issues.apache.org/jira/secure/attachment/12945103/HIVE-20772.02.patch
--jira HIVE-20772
Build timed out (after 500 minutes). Marking the build as aborted.
{noformat}
no test reports... not sure what's up with ptest
> record per-task CPU counters in LLAP
>
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.01.patch, HIVE-20772.02.patch,
> HIVE-20772.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20776) Move HMS filterHooks from client-side to server-side
[ https://issues.apache.org/jira/browse/HIVE-20776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661367#comment-16661367 ] Vihang Karajgaonkar commented on HIVE-20776: Not when impersonation is turned OFF. It will always show UGI as hive in that case AFAIK > Move HMS filterHooks from client-side to server-side > > > Key: HIVE-20776 > URL: https://issues.apache.org/jira/browse/HIVE-20776 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Karthik Manamcheri >Assignee: Karthik Manamcheri >Priority: Major > > In HMS, I noticed that all the filter hooks are applied on the client side > (in HiveMetaStoreClient.java). Is there any reason why we can't apply the > filters on the server-side? > Motivation: Some newer apache projects such as Kudu use HMS for metadata > storage. Kudu is not completely Java-based and there are interaction points > where they have C++ clients. In such cases, it would be ideal to have > consistent behavior from HMS side as far as filters, etc are concerned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20791) Enforce a "Spill Maximum" per query
[ https://issues.apache.org/jira/browse/HIVE-20791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661353#comment-16661353 ] Gopal V commented on HIVE-20791: The new workload management rules let you set his limit up by user/resource pool limits. > Enforce a "Spill Maximum" per query > --- > > Key: HIVE-20791 > URL: https://issues.apache.org/jira/browse/HIVE-20791 > Project: Hive > Issue Type: New Feature > Components: Configuration >Reporter: Peter Ebert >Priority: Major > > Currently one query that spills excessively to disk can use up all cluster > space and cause other queries/workloads to fail. I understand that a global > maximum would be difficult to track, but some limit per mapper/reducer would > help keep the spill under control. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20792) Inserting timestamp with zones truncates the data
[
https://issues.apache.org/jira/browse/HIVE-20792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M reassigned HIVE-20792:
--
Assignee: Jaume M
> Inserting timestamp with zones truncates the data
> -
>
> Key: HIVE-20792
> URL: https://issues.apache.org/jira/browse/HIVE-20792
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
>
> For example with the table:
> {code}
> CREATE TABLE myTable
> (
> a TIMESTAMP
> )
> STORED AS ORC
> tblproperties("transactional"="true");
> {code}
> The following inserts store the wrong data:
> {code}
> INSERT INTO myTable VALUES("2018-10-19 10:35:00 UTC"); -> 2018-10-19
> 00:00:00.0
> INSERT INTO myTable VALUES("2018-10-19 10:35:00 ZZZ"); -> 2018-10-19
> 00:00:00.0
> {code}
> The second one should fail since ZZZ is not a time zone.
> Similarly if the column is of type DATE,
> {code}
> INSERT INTO myTableDate VALUES("2018-10-19 "); -> 2018-10-19
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20776) Move HMS filterHooks from client-side to server-side
[ https://issues.apache.org/jira/browse/HIVE-20776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661303#comment-16661303 ] Karthik Manamcheri commented on HIVE-20776: --- [~vihangk1] hmm This should work if we use Kerberos and pick up the UGI information from the environment yes? > Move HMS filterHooks from client-side to server-side > > > Key: HIVE-20776 > URL: https://issues.apache.org/jira/browse/HIVE-20776 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Karthik Manamcheri >Assignee: Karthik Manamcheri >Priority: Major > > In HMS, I noticed that all the filter hooks are applied on the client side > (in HiveMetaStoreClient.java). Is there any reason why we can't apply the > filters on the server-side? > Motivation: Some newer apache projects such as Kudu use HMS for metadata > storage. Kudu is not completely Java-based and there are interaction points > where they have C++ clients. In such cases, it would be ideal to have > consistent behavior from HMS side as far as filters, etc are concerned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661288#comment-16661288
]
Daniel Dai commented on HIVE-18767:
---
Unlink from 3.1.1/2.4.4 as they are about to release.
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 3.2.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch,
> HIVE-18767.2-branch-3.1.patch, HIVE-18767.2.branch-3.patch,
> HIVE-18767.2.patch, HIVE-18767.3-branch-3.1.patch,
> HIVE-18767.3.branch-3.patch, HIVE-18767.3.patch,
> HIVE-18767.4-branch-3.1.patch, HIVE-18767.4.branch-3.1.patch,
> HIVE-18767.4.patch, HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-18767:
--
Fix Version/s: (was: 2.3.4)
(was: 3.1.1)
3.2.0
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 3.2.0
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch,
> HIVE-18767.2-branch-3.1.patch, HIVE-18767.2.branch-3.patch,
> HIVE-18767.2.patch, HIVE-18767.3-branch-3.1.patch,
> HIVE-18767.3.branch-3.patch, HIVE-18767.3.patch,
> HIVE-18767.4-branch-3.1.patch, HIVE-18767.4.branch-3.1.patch,
> HIVE-18767.4.patch, HIVE-18767.5.patch, HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661281#comment-16661281
]
Daniel Dai commented on HIVE-18778:
---
Further make a patch for branch-2. This takes a shortcut to avoid mass golden
file updates.
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0, 3.2.0, 3.1.1
>
> Attachments: HIVE-18778-SparkPositive.patch,
> HIVE-18778.1.branch-2.patch, HIVE-18778.1.patch,
> HIVE-18778.10.branch-3.patch, HIVE-18778.11.branch-3.1.patch,
> HIVE-18778.11.branch-3.patch, HIVE-18778.12.branch-3.1.patch,
> HIVE-18778.2.patch, HIVE-18778.3.patch, HIVE-18778.4.patch,
> HIVE-18778.5.patch, HIVE-18778.6.patch, HIVE-18778.7.patch,
> HIVE-18778.8.patch, HIVE-18778.9.branch-3.patch, HIVE-18778.9.patch,
> HIVE-18778_TestCliDriver.patch, HIVE-18788_SparkNegative.patch,
> HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-18778:
--
Attachment: HIVE-18778.1.branch-2.patch
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0, 3.2.0, 3.1.1
>
> Attachments: HIVE-18778-SparkPositive.patch,
> HIVE-18778.1.branch-2.patch, HIVE-18778.1.patch,
> HIVE-18778.10.branch-3.patch, HIVE-18778.11.branch-3.1.patch,
> HIVE-18778.11.branch-3.patch, HIVE-18778.12.branch-3.1.patch,
> HIVE-18778.2.patch, HIVE-18778.3.patch, HIVE-18778.4.patch,
> HIVE-18778.5.patch, HIVE-18778.6.patch, HIVE-18778.7.patch,
> HIVE-18778.8.patch, HIVE-18778.9.branch-3.patch, HIVE-18778.9.patch,
> HIVE-18778_TestCliDriver.patch, HIVE-18788_SparkNegative.patch,
> HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20737) Local SparkContext is shared between user sessions and should be closed only when there is no active
[ https://issues.apache.org/jira/browse/HIVE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661206#comment-16661206 ] Sahil Takiar commented on HIVE-20737: - Filed HIVE-20790 to fix the issue where we want to call close while open is running. > Local SparkContext is shared between user sessions and should be closed only > when there is no active > > > Key: HIVE-20737 > URL: https://issues.apache.org/jira/browse/HIVE-20737 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-20737.1.patch, HIVE-20737.10.patch, > HIVE-20737.11.patch, HIVE-20737.12.patch, HIVE-20737.2.patch, > HIVE-20737.5.patch, HIVE-20737.6.patch, HIVE-20737.7.patch, > HIVE-20737.8.patch, HIVE-20737.9.patch > > > 1. Local SparkContext is shared between user sessions and should be closed > only when there is no active. > 2. Possible race condition in SparkSession.open() in case when user queries > run in parallel within the same session. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20519) Remove 30m min value for hive.spark.session.timeout
[
https://issues.apache.org/jira/browse/HIVE-20519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661203#comment-16661203
]
Sahil Takiar commented on HIVE-20519:
-
Aforementioned issue will be fixed in HIVE-20790
> Remove 30m min value for hive.spark.session.timeout
> ---
>
> Key: HIVE-20519
> URL: https://issues.apache.org/jira/browse/HIVE-20519
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20519.1.patch, HIVE-20519.2.patch
>
>
> In HIVE-14162 we added the config \{{hive.spark.session.timeout}} which
> provided a way to time out Spark sessions that are active for a long period
> of time. The config has a lower bound of 30m which we should remove. It
> should be possible for users to configure this value so the HoS session is
> closed as soon as the query is complete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-14898) HS2 shouldn't log callstack for an empty auth header error
[
https://issues.apache.org/jira/browse/HIVE-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661093#comment-16661093
]
Kevin Risden edited comment on HIVE-14898 at 10/23/18 6:24 PM:
---
I think this is a bandaid for actually addressing the root cause. The way the
code flows is that there is a try/catch for something that HS2 can short
circuit a lot sooner.
An example of this improvement in HBase from HBASE-19852:
[https://github.com/apache/hbase/blob/master/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java#L94]
This could be done much earlier in the code flow here:
[https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L155]
was (Author: risdenk):
I think this is a bandaid for actually addressing the root cause. The way the
code flows is that there is a try/catch for something that HS2 can short
circuit a lot sooner.
An example of this improvement in HBase:
[https://github.com/apache/hbase/blob/master/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java#L94]
This could be done much earlier in the code flow here:
[https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L155]
> HS2 shouldn't log callstack for an empty auth header error
> --
>
> Key: HIVE-14898
> URL: https://issues.apache.org/jira/browse/HIVE-14898
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-14898.1.patch
>
>
> Currently when the auth header is not sent by the client (Knox seems to do
> this every time - it only adds auth header after receiving 401), HS2 logs the
> following twice, for two principals.
> The callstack is useless because this is an expected condition and 401 is
> returned to the client.
> {noformat}
> 2016-10-05 15:32:02,408 ERROR [HiveServer2-HttpHandler-Pool: Thread-199]:
> thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(169)) -
> Failed to authenticate with hive/_HOST kerberos principal
> 2016-10-05 15:32:02,408 ERROR [HiveServer2-HttpHandler-Pool: Thread-199]:
> thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(104)) - Error:
> org.apache.hive.service.auth.HttpAuthenticationException:
> java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:170)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:83)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> at org.eclipse.jetty.server.Server.handle(Server.java:349)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
> at
>
[jira] [Commented] (HIVE-14898) HS2 shouldn't log callstack for an empty auth header error
[
https://issues.apache.org/jira/browse/HIVE-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661093#comment-16661093
]
Kevin Risden commented on HIVE-14898:
-
I think this is a bandaid for actually addressing the root cause. The way the
code flows is that there is a try/catch for something that HS2 can short
circuit a lot sooner.
An example of this improvement in HBase:
[https://github.com/apache/hbase/blob/master/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java#L94]
This could be done much earlier in the code flow here:
[https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L155]
> HS2 shouldn't log callstack for an empty auth header error
> --
>
> Key: HIVE-14898
> URL: https://issues.apache.org/jira/browse/HIVE-14898
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-14898.1.patch
>
>
> Currently when the auth header is not sent by the client (Knox seems to do
> this every time - it only adds auth header after receiving 401), HS2 logs the
> following twice, for two principals.
> The callstack is useless because this is an expected condition and 401 is
> returned to the client.
> {noformat}
> 2016-10-05 15:32:02,408 ERROR [HiveServer2-HttpHandler-Pool: Thread-199]:
> thrift.ThriftHttpServlet (ThriftHttpServlet.java:doKerberosAuth(169)) -
> Failed to authenticate with hive/_HOST kerberos principal
> 2016-10-05 15:32:02,408 ERROR [HiveServer2-HttpHandler-Pool: Thread-199]:
> thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(104)) - Error:
> org.apache.hive.service.auth.HttpAuthenticationException:
> java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:170)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:83)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> at org.eclipse.jetty.server.Server.handle(Server.java:349)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:167)
> ... 23 more
> Caused by: org.apache.hive.service.auth.HttpAuthenticationException:
> Authorization header received from the client is empty.
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.getAuthHeader(ThriftHttpServlet.java:311)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.access$100(ThriftHttpServlet.java:59)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:212)
> at
>
[jira] [Updated] (HIVE-20789) HiveServer2 should have Timeouts against clients that never close sockets
[
https://issues.apache.org/jira/browse/HIVE-20789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20789:
-
Status: Patch Available (was: Open)
We noticed that the TSocket used by the thrift thread pools are purposely
initiated with 0 clientTimeout. It makes sense to make it configurable to
prevent DDOS.
> HiveServer2 should have Timeouts against clients that never close sockets
> -
>
> Key: HIVE-20789
> URL: https://issues.apache.org/jira/browse/HIVE-20789
> Project: Hive
> Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Attachments: HIVE-20789.patch
>
>
> We have had a scenario that health checks sending 0 bytes to HiveServer2
> sockets would DDOS the HiveServer2, if for some reason they hang or otherwise
> don't send TCP FIN, then all HiveServer2 thrift thread-pool threads will
> block reading the socket.
> This is the stack (we are running an older version of Hive here)
> {noformat}
> "HiveServer2-Handler-Pool: Thread-2512239" - Thread t@2512239
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <23781b74> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
> at
> org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Eventually HiveServer2 has no more free threads left.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20789) HiveServer2 should have Timeouts against clients that never close sockets
[
https://issues.apache.org/jira/browse/HIVE-20789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20789:
-
Attachment: HIVE-20789.patch
> HiveServer2 should have Timeouts against clients that never close sockets
> -
>
> Key: HIVE-20789
> URL: https://issues.apache.org/jira/browse/HIVE-20789
> Project: Hive
> Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Attachments: HIVE-20789.patch
>
>
> We have had a scenario that health checks sending 0 bytes to HiveServer2
> sockets would DDOS the HiveServer2, if for some reason they hang or otherwise
> don't send TCP FIN, then all HiveServer2 thrift thread-pool threads will
> block reading the socket.
> This is the stack (we are running an older version of Hive here)
> {noformat}
> "HiveServer2-Handler-Pool: Thread-2512239" - Thread t@2512239
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <23781b74> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
> at
> org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Eventually HiveServer2 has no more free threads left.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20789) HiveServer2 should have Timeouts against clients that never close sockets
[
https://issues.apache.org/jira/browse/HIVE-20789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned HIVE-20789:
Assignee: Szehon Ho
> HiveServer2 should have Timeouts against clients that never close sockets
> -
>
> Key: HIVE-20789
> URL: https://issues.apache.org/jira/browse/HIVE-20789
> Project: Hive
> Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
>
> We have had a scenario that health checks sending 0 bytes to HiveServer2
> sockets would DDOS the HiveServer2, if for some reason they hang or otherwise
> don't send TCP FIN, then all HiveServer2 thrift thread-pool threads will
> block reading the socket.
> This is the stack (we are running an older version of Hive here)
> {noformat}
> "HiveServer2-Handler-Pool: Thread-2512239" - Thread t@2512239
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <23781b74> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
> at
> org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Eventually HiveServer2 has no more free threads left.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20789) HiveServer2 should have Timeouts against clients that never close sockets
[
https://issues.apache.org/jira/browse/HIVE-20789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20789:
-
Description:
We have had a scenario that health checks sending 0 bytes to HiveServer2
sockets would DDOS the HiveServer2, if for some reason they hang or otherwise
don't send TCP FIN, then all HiveServer2 thrift thread-pool threads will block
reading the socket.
This is the stack (we are running an older version of Hive here)
{noformat}
"HiveServer2-Handler-Pool: Thread-2512239" - Thread t@2512239
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <23781b74> (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
at
org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){noformat}
Eventually HiveServer2 has no more free threads left.
was:
We have had a scenario that health checks sending 0 bytes to HiveServer2
sockets would DDOS the HiveServer2, if they dont send TCP FIN then they will
continually cause all HiveServer2 thrift thread-pool threads to block at this
stack (we are running an older version of Hive here, so ignore the lines)
{noformat}
"HiveServer2-Handler-Pool: Thread-2512239" - Thread t@2512239
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <23781b74> (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
at
org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){noformat}
Eventually HiveServer2 has no more free threads left.
> HiveServer2 should have Timeouts against clients that never close sockets
> -
>
> Key: HIVE-20789
>
[jira] [Assigned] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned HIVE-20786:
Assignee: Szehon Ho
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIPPED
> [INFO] Hive
[jira] [Updated] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20786:
-
Attachment: HIVE-20786.patch
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIPPED
> [INFO] Hive Metastore Tools common libraries
[jira] [Updated] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20786:
-
Status: Patch Available (was: Open)
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIPPED
> [INFO]
[jira] [Commented] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660999#comment-16660999
]
Szehon Ho commented on HIVE-20786:
--
Yes I've hit this error for awhile now. Uploading a patch that seems to solve
it for me. Though I wasn't following so not sure why it was set on purpose to
gnu before.
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Priority: Major
> Labels: maven
> Attachments: hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging
[jira] [Commented] (HIVE-4134) Facing issues while executing commands on hive shell. The system throws following error: only on Windows Cygwin setup
[ https://issues.apache.org/jira/browse/HIVE-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660939#comment-16660939 ] Cassie Griffin commented on HIVE-4134: -- Hive shell is a good tool to use for SQL client on Hive server. For this, You have to use the latest version of the window. I have found the same problem on [https://babasupport.org/windows/windows-error-0x8024200b|https://babasupport.org/windows/windows-error-0x8024200b/]/ You can ask your problem for a good solution > Facing issues while executing commands on hive shell. The system throws > following error: only on Windows Cygwin setup > -- > > Key: HIVE-4134 > URL: https://issues.apache.org/jira/browse/HIVE-4134 > Project: Hive > Issue Type: Task > Environment: hadop1.1.1 cygwin hive-0.8.1 >Reporter: Avinash Singh >Priority: Major > > When executing this command > $ bin/hive -e 'SHOW TABLES' > WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use > org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. > Logging initialized using configuration in > jar:file:/C:/cygwin/home/Administrator/hive-0.8.1/lib/hive-common-0.8.1.jar!/hive-log4j.properties > Hive history > file=/tmp/Administrator/hive_job_log_Administrator_201303071139_1504829167.txt > Patch for HADOOP-7682: Instantiating workaround file system > FAILED: Hive Internal Error: > java.lang.IllegalArgumentException(java.net.URISyntaxException: Relative path > in absolute URI: > file:C:/cygwin/tmp//Administrator/hive_2013-03-07_11-39-26_427_243610179348269546) > java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative > path in absolute URI: > file:C:/cygwin/tmp//Administrator/hive_2013-03-07_11-39-26_427_243610179348269546 > at org.apache.hadoop.fs.Path.initialize(Path.java:148) > at org.apache.hadoop.fs.Path.(Path.java:132) > at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:160) > at > org.apache.hadoop.hive.ql.Context.getLocalScratchDir(Context.java:187) > at > org.apache.hadoop.hive.ql.Context.getLocalTmpFileURI(Context.java:303) > at > org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:227) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:243) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:889) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:338) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:637) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > Caused by: java.net.URISyntaxException: Relative path in absolute URI: > file:C:/cygwin/tmp//Administrator/hive_2013-03-07_11-39-26_427_243610179348269546 > at java.net.URI.checkPath(URI.java:1788) > at java.net.URI.(URI.java:734) > at org.apache.hadoop.fs.Path.initialize(Path.java:145) > ... 20 more > tried using the bug fix as describe in > https://issues.apache.org/jira/browse/HIVE-2388 but his also not worked > bin/start.sh > WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use > org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. > Exception in thread "main" java.lang.RuntimeException: Failed to initialize > default Hive configuration variables! > at org.apache.hadoop.hive.conf.HiveConf.getConfVarURL(HiveConf.java:651) > at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:815) > at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:776) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:576) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at >
[jira] [Updated] (HIVE-20788) Extended SJ reduction may backtrack columns incorrectly when creating filters
[ https://issues.apache.org/jira/browse/HIVE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20788: --- Attachment: HIVE-20788.02.patch > Extended SJ reduction may backtrack columns incorrectly when creating filters > - > > Key: HIVE-20788 > URL: https://issues.apache.org/jira/browse/HIVE-20788 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20788.01.patch, HIVE-20788.02.patch, > HIVE-20788.patch > > > Affects TPCDS query 24 with constraints. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20542) Incremental REPL DUMP progress information log message is incorrect.
[ https://issues.apache.org/jira/browse/HIVE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20542: Resolution: Fixed Fix Version/s: 4.0.0 Target Version/s: 4.0.0 (was: 4.0.0, 3.2.0) Status: Resolved (was: Patch Available) 04.patch is committed to master! Thanks [~ashutosh.bapat] for the patch! > Incremental REPL DUMP progress information log message is incorrect. > > > Key: HIVE-20542 > URL: https://issues.apache.org/jira/browse/HIVE-20542 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0, 3.2.0 >Reporter: Sankar Hariappan >Assignee: Ashutosh Bapat >Priority: Minor > Labels: DR, Replication, pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20542.01.patch, HIVE-20542.02, HIVE-20542.03, > HIVE-20542.04 > > > Incremental REPL DUMP have the progress information logged as > "eventsDumpProgress":"49/0". > It should actually log the estimated number of events are denominator but it > is coming as 0 always. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20542) Incremental REPL DUMP progress information log message is incorrect.
[ https://issues.apache.org/jira/browse/HIVE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660554#comment-16660554 ] Sankar Hariappan commented on HIVE-20542: - +1 > Incremental REPL DUMP progress information log message is incorrect. > > > Key: HIVE-20542 > URL: https://issues.apache.org/jira/browse/HIVE-20542 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0, 3.2.0 >Reporter: Sankar Hariappan >Assignee: Ashutosh Bapat >Priority: Minor > Labels: DR, Replication, pull-request-available > Attachments: HIVE-20542.01.patch, HIVE-20542.02, HIVE-20542.03, > HIVE-20542.04 > > > Incremental REPL DUMP have the progress information logged as > "eventsDumpProgress":"49/0". > It should actually log the estimated number of events are denominator but it > is coming as 0 always. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20682:
Status: Patch Available (was: Open)
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> We shall maintain an Atomic reference counter in the Hive object. We
> increment the counter when somebody sets it in thread local hiveDb and
> decrement it when somebody releases it. Only when the counter is down to 0,
> we should close the connection.
> Couple of cases to release the thread local hiveDb:
> * When synchronous query execution in master thread re-creates Hive object
> due to config change. We also, need to update the sessionHive object in the
> current session as we releases it from thread local hiveDb of master thread.
> * When async thread exits after completing execution or due to exception.
> If the session is getting closed, it is guaranteed that the reference counter
> is down to 0 as we forcefully close all the async operations and so we can
> close the connection there.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20682:
Attachment: HIVE-20682.01.patch
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> We shall maintain an Atomic reference counter in the Hive object. We
> increment the counter when somebody sets it in thread local hiveDb and
> decrement it when somebody releases it. Only when the counter is down to 0,
> we should close the connection.
> Couple of cases to release the thread local hiveDb:
> * When synchronous query execution in master thread re-creates Hive object
> due to config change. We also, need to update the sessionHive object in the
> current session as we releases it from thread local hiveDb of master thread.
> * When async thread exits after completing execution or due to exception.
> If the session is getting closed, it is guaranteed that the reference counter
> is down to 0 as we forcefully close all the async operations and so we can
> close the connection there.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20682:
Attachment: (was: HIVE-20682.01.patch)
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> We shall maintain an Atomic reference counter in the Hive object. We
> increment the counter when somebody sets it in thread local hiveDb and
> decrement it when somebody releases it. Only when the counter is down to 0,
> we should close the connection.
> Couple of cases to release the thread local hiveDb:
> * When synchronous query execution in master thread re-creates Hive object
> due to config change. We also, need to update the sessionHive object in the
> current session as we releases it from thread local hiveDb of master thread.
> * When async thread exits after completing execution or due to exception.
> If the session is getting closed, it is guaranteed that the reference counter
> is down to 0 as we forcefully close all the async operations and so we can
> close the connection there.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20682:
Status: Open (was: Patch Available)
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> We shall maintain an Atomic reference counter in the Hive object. We
> increment the counter when somebody sets it in thread local hiveDb and
> decrement it when somebody releases it. Only when the counter is down to 0,
> we should close the connection.
> Couple of cases to release the thread local hiveDb:
> * When synchronous query execution in master thread re-creates Hive object
> due to config change. We also, need to update the sessionHive object in the
> current session as we releases it from thread local hiveDb of master thread.
> * When async thread exits after completing execution or due to exception.
> If the session is getting closed, it is guaranteed that the reference counter
> is down to 0 as we forcefully close all the async operations and so we can
> close the connection there.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20682:
Status: Patch Available (was: Open)
[~pvary], [~daijy]
Can you please take a look at the patch?
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Attachments: HIVE-20682.01.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> We shall maintain an Atomic reference counter in the Hive object. We
> increment the counter when somebody sets it in thread local hiveDb and
> decrement it when somebody releases it. Only when the counter is down to 0,
> we should close the connection.
> Couple of cases to release the thread local hiveDb:
> * When synchronous query execution in master thread re-creates Hive object
> due to config change. We also, need to update the sessionHive object in the
> current session as we releases it from thread local hiveDb of master thread.
> * When async thread exits after completing execution or due to exception.
> If the session is getting closed, it is guaranteed that the reference counter
> is down to 0 as we forcefully close all the async operations and so we can
> close the connection there.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660488#comment-16660488
]
ASF GitHub Bot commented on HIVE-20682:
---
GitHub user sankarh opened a pull request:
https://github.com/apache/hive/pull/452
HIVE-20682: Async query execution can potentially fail if shared
sessionHive is closed by master thread.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sankarh/hive HIVE-20682
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hive/pull/452.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #452
commit 9f5a3eff3598d7b76fd7a53f6aedefd57e1c641e
Author: Sankar Hariappan
Date: 2018-10-06T08:12:25Z
HIVE-20682: Async query execution can potentially fail if shared
sessionHive is closed by master thread.
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> We shall maintain an Atomic reference counter in the Hive object. We
> increment the counter when somebody sets it in thread local hiveDb and
> decrement it when somebody releases it. Only when the counter is down to 0,
> we should close the connection.
> Couple of cases to release the thread local hiveDb:
> * When synchronous query execution in master thread re-creates Hive object
> due to config change. We also, need to update the sessionHive object in the
> current session as we releases it from thread local hiveDb of master thread.
> * When async thread exits after completing execution or due to exception.
> If the session is getting closed, it is guaranteed that the reference counter
> is down to 0 as we forcefully close all the async operations and so we can
> close the connection there.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-20682:
--
Labels: pull-request-available (was: )
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> We shall maintain an Atomic reference counter in the Hive object. We
> increment the counter when somebody sets it in thread local hiveDb and
> decrement it when somebody releases it. Only when the counter is down to 0,
> we should close the connection.
> Couple of cases to release the thread local hiveDb:
> * When synchronous query execution in master thread re-creates Hive object
> due to config change. We also, need to update the sessionHive object in the
> current session as we releases it from thread local hiveDb of master thread.
> * When async thread exits after completing execution or due to exception.
> If the session is getting closed, it is guaranteed that the reference counter
> is down to 0 as we forcefully close all the async operations and so we can
> close the connection there.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20682:
Attachment: HIVE-20682.01.patch
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Attachments: HIVE-20682.01.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> We shall maintain an Atomic reference counter in the Hive object. We
> increment the counter when somebody sets it in thread local hiveDb and
> decrement it when somebody releases it. Only when the counter is down to 0,
> we should close the connection.
> Couple of cases to release the thread local hiveDb:
> * When synchronous query execution in master thread re-creates Hive object
> due to config change. We also, need to update the sessionHive object in the
> current session as we releases it from thread local hiveDb of master thread.
> * When async thread exits after completing execution or due to exception.
> If the session is getting closed, it is guaranteed that the reference counter
> is down to 0 as we forcefully close all the async operations and so we can
> close the connection there.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660362#comment-16660362
]
Zoltan Haindrich commented on HIVE-20617:
-
#10 will also address some more correctness issues
* during in comparision numeric might not have been compared as double as it
should.
* mixed numeric and string comparisions in IN: may have missed one either
string or numeric results:
{code}
select * from t where a in (1.0,'x',2);
{code}
* struct containing null might have not compared correctly
{code}
select 'expected 2',count(*) from ax where (s,t) in (('a','a'),(null, 'bb'));
{code}
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch,
> HIVE-20617.11.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20617:
Attachment: HIVE-20617.11.patch
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch,
> HIVE-20617.11.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20617:
Attachment: HIVE-20617.10.patch
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20617:
Attachment: HIVE-20617.10.patch
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20779) Fix missing results when IN operator has a struct with a null field
[
https://issues.apache.org/jira/browse/HIVE-20779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660233#comment-16660233
]
Igor Kryvenko commented on HIVE-20779:
--
[~kgyrtkirk] Okay, let me know if I can help you with that issue.
> Fix missing results when IN operator has a struct with a null field
> ---
>
> Key: HIVE-20779
> URL: https://issues.apache.org/jira/browse/HIVE-20779
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Igor Kryvenko
>Priority: Major
>
> {code}
> create table ax(s char(1),t char(10));
> insert into ax values ('a','a'),('a','a '),('b','bb');
> select 'expected 1',count(*) from ax where ((s,t) in (('a','a'),(null,
> 'bb'))) is null;
> -- returns 0 rows
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20779) Fix missing results when IN operator has a struct with a null field
[
https://issues.apache.org/jira/browse/HIVE-20779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660221#comment-16660221
]
Zoltan Haindrich commented on HIVE-20779:
-
[~ikryvenko] latest version of HIVE-20617 will probably also address this -
alongside with another IN related resultset issue
> Fix missing results when IN operator has a struct with a null field
> ---
>
> Key: HIVE-20779
> URL: https://issues.apache.org/jira/browse/HIVE-20779
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Igor Kryvenko
>Priority: Major
>
> {code}
> create table ax(s char(1),t char(10));
> insert into ax values ('a','a'),('a','a '),('b','bb');
> select 'expected 1',count(*) from ax where ((s,t) in (('a','a'),(null,
> 'bb'))) is null;
> -- returns 0 rows
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-2927) Allow escape character in get_json_object
[
https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660136#comment-16660136
]
Michael Chirico commented on HIVE-2927:
---
workaround is to use lateral view json_tuple
select cdotd
from my_tbl
lateral view json_tuple(json, ''c.d") J as cdotd
> Allow escape character in get_json_object
> -
>
> Key: HIVE-2927
> URL: https://issues.apache.org/jira/browse/HIVE-2927
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 0.8.1, 0.9.0
>Reporter: Sean McNamara
>Priority: Major
> Attachments: HIVE-2927.1.patch.txt
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> *Background:*
> get_json_object extracts json objects from a json string based on a specified
> path.
> *Problem:*
> The current implementation of get_json_object can't see keys with a '.' in
> them. Our data contains '.' in the keys, so we have to filter our json keys
> through a streaming script to replace '.' for '_'.
> *Example:*
> {{json = {"a":{"b": 1}, "c.d": 2}}}
> {{get_json_object(json, "$.a.b") returns: 1}}
> {{get_json_object(json, "$.c.d") returns: NULL}}
> In the present implementation of get_json_object, c.d is not addressable.
> *Proposal:*
> The desired behavior would be to allow the JSON path to be escape-able, like
> so:
> {{get_json_object(json, '$.c\\\.d') would return: 2}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
