[jira] [Created] (HIVE-21829) HiveMetaStore authorization with AlterTable and DropTable events

[Ramesh Mani (JIRA)]

Ramesh Mani created HIVE-21829:
--

 Summary: HiveMetaStore authorization with AlterTable and DropTable 
events
 Key: HIVE-21829
 URL: https://issues.apache.org/jira/browse/HIVE-21829
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.1.2
Reporter: Ramesh Mani
 Fix For: 3.1.2


With HIVE-21753, we have HiveMetastore authorizer which uses HiveAuthorizer 
interface to authorizer metastore events.
This jira is to fix a bug in HIVE-21753 which failed to authorizer Alter and 
DropTable events




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21828) Tez: Use a TezStaticConfiguration inherited from TezRuntimeConfiguration

[Gopal V (JIRA)]

Gopal V created HIVE-21828:
--

 Summary: Tez: Use a TezStaticConfiguration inherited from 
TezRuntimeConfiguration
 Key: HIVE-21828
 URL: https://issues.apache.org/jira/browse/HIVE-21828
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


The HS2 tez-site.xml does not change dynamically - the XML parsed components of 
the config can be obtained statically and kept across sessions.

This allows for the replacing of "new TezConfiguration()" with a HS2 local 
version instead.

The configuration object however has to reference the right resource file (i.e 
location of tez-site.xml) without reparsing it for each query.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21827) Multiple calls in Semantic

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-21827:
--

 Summary: Multiple calls in Semantic
 Key: HIVE-21827
 URL: https://issues.apache.org/jira/browse/HIVE-21827
 Project: Hive
  Issue Type: Bug
Reporter: Jesus Camacho Rodriguez






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21826) Backport HIVE-21786 to branch-3

[Alan Gates (JIRA)]

Alan Gates created HIVE-21826:
-

 Summary: Backport HIVE-21786 to branch-3
 Key: HIVE-21826
 URL: https://issues.apache.org/jira/browse/HIVE-21826
 Project: Hive
  Issue Type: Bug
Reporter: Alan Gates
Assignee: Alan Gates


Missed branch-3 in the original fix.  Need to apply the patch here as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21825) Improve client error msg when Active/Passive HA is enabled

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-21825:


 Summary: Improve client error msg when Active/Passive HA is enabled
 Key: HIVE-21825
 URL: https://issues.apache.org/jira/browse/HIVE-21825
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0, 3.2.0
Reporter: Prasanth Jayachandran


When Active/Passive HA is enabled and when client tries to connect to Passive 
HA or when HS2 is still starting up, clients will receive the following the 
error msg
{code:java}
'Cannot open sessions on an inactive HS2 instance; use service discovery to 
connect'{code}
This error msg can be improved to say that HS2 is still starting up (or more 
user-friendly error msg). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70770: Break up DDLTask - extract Storage related operations

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70770/#review215643
---




ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableSetSerdeOperation.java
Lines 68 (patched)


this is odd
could you open a followup ticket?


- Zoltan Haindrich


On June 1, 2019, 11:03 a.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70770/
> ---
> 
> (Updated June 1, 2019, 11:03 a.m.)
> 
> 
> Review request for hive and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-21797
> https://issues.apache.org/jira/browse/HIVE-21797
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is 
> also a huge class, which has a field for each DDL operation it supports. The 
> goal is to refactor these in order to have everything cut into more 
> handleable classes under the package  org.apache.hadoop.hive.ql.exec.ddl:
> 
> have a separate class for each operation
> have a package for each operation group (database ddl, table ddl, etc), so 
> the amount of classes under a package is more manageable
> make all the requests (DDLDesc subclasses) immutable
> DDLTask should be agnostic to the actual operations
> right now let's ignore the issue of having some operations handled by DDLTask 
> which are not actual DDL operations (lock, unlock, desc...)
> In the interim time when there are two DDLTask and DDLWork classes in the 
> code base the new ones in the new package are called DDLTask2 and DDLWork2 
> thus avoiding the usage of fully qualified class names where both the old and 
> the new classes are in use.
> 
> Step #10: extract all the storage related operations from the old DDLTask, 
> and move them under the new package.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java
>  abd351d51a 
>   ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractAlterTableDesc.java 
> 3ff04131d0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractAlterTableOperation.java
>  5d8cd94939 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractAlterTableWithConstraintsDesc.java
>  ebd3bdbf4d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/AlterTableAddColumnsDesc.java
>  a15597c071 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/AlterTableChangeColumnDesc.java
>  827cc80d14 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/AlterTableReplaceColumnsDesc.java
>  6947c1ed51 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/AlterTableReplaceColumnsOperation.java
>  116fa2d28c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/column/AlterTableUpdateColumnsDesc.java
>  f1a1ea12e5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/constaint/AlterTableAddConstraintDesc.java
>  cd8deab596 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/creation/ShowCreateTableOperation.java
>  3ea7443787 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableClusteredByDesc.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableClusteredByOperation.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableIntoBucketsDesc.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableIntoBucketsOperation.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotClusteredDesc.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotClusteredOperation.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotSkewedDesc.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotSkewedOperation.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotSortedDesc.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableNotSortedOperation.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableSetFileFormatDesc.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableSetFileFormatOperation.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableSetLocationDesc.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableSetLocationOperation.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/Alte

[jira] [Created] (HIVE-21824) Weird output format of 'describe formatted' command

[JIRA]

Zoltán Borók-Nagy created HIVE-21824:


 Summary: Weird output format of 'describe formatted' command
 Key: HIVE-21824
 URL: https://issues.apache.org/jira/browse/HIVE-21824
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.1.0
Reporter: Zoltán Borók-Nagy


Hive 3 changed the output format of the 'describe formatted' command. The new 
format is vertical instead of the old horizontal format.

The expected output can be looked at the tests, e.g.:

[https://github.com/apache/hive/blob/10b6d70da1442cccf533bc97f56a622ec9f39661/ql/src/test/results/clientpositive/describe_table.q.out#L201-L216]

However, for Hive 3.1.0 (and probably for other Hive versions as well) the 
output looks really strange:
{noformat}
jdbc:hive2://localhost:11050/default> describe formatted test i;
+++---+---++-+--+--++-++--+
| col_name | data_type | min | max | num_nulls | distinct_count | avg_col_len | 
max_col_len | num_trues | num_falses | bitvector | comment |
+++---+---++-+--+--++-++--+
| col_name | i | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 
NULL |
| data_type | int | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 
NULL | NULL |
| min | 1 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL 
|
| max | 3 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL 
|
| num_nulls | 0 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL 
| NULL |
| distinct_count | 3 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 
NULL | NULL |
| avg_col_len | | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL 
| NULL |
| max_col_len | | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL 
| NULL |
| num_trues | | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 
NULL |
| num_falses | | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 
NULL |
| bitVector | | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 
NULL |
| comment | from deserializer | NULL | NULL | NULL | NULL | NULL | NULL | NULL 
| NULL | NULL | NULL |
| COLUMN_STATS_ACCURATE | 
{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"i\":\"true\"}} | NULL | NULL | 
NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL |
+++---+---++-+--+--++-++--+{noformat}
Notice that the attribute names in the first column are the same as the column 
names. And all the attribute values are listed in the 'data_type' column.

On cwiki there is also another example to this weird output at 
[https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ExistingTables],
 search for "the output would look like this". But it doesn't seem intentional 
to me.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21823) New metrics to get the average queue length / free executor number for a given time window

[Peter Vary (JIRA)]

Peter Vary created HIVE-21823:
-

 Summary: New metrics to get the average queue length / free 
executor number for a given time window
 Key: HIVE-21823
 URL: https://issues.apache.org/jira/browse/HIVE-21823
 Project: Hive
  Issue Type: Sub-task
  Components: llap
Reporter: Peter Vary
Assignee: Peter Vary


We need to calculate the average queue size / free executor size for a window 
to have good data for making routing decisions.

Interesting things to consider:
 * The time gap between arriving request can be different, so simple average is 
not enough to have correct data
 * We need to have 2 parameters
 ** Time window length
 ** Maximum data point numbers - so we will not collect "infinite" amount of 
data at high load



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21822) Expose LlapDaemon metrics through a new API method

[Peter Vary (JIRA)]

Peter Vary created HIVE-21822:
-

 Summary: Expose LlapDaemon metrics through a new API method
 Key: HIVE-21822
 URL: https://issues.apache.org/jira/browse/HIVE-21822
 Project: Hive
  Issue Type: Sub-task
  Components: llap
Reporter: Peter Vary
Assignee: Peter Vary


We should add a new API method to access specific LlapDaemon metrics which 
would provide information to TezAMs to make good decisions on load balancing 
tasks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)