[jira] [Created] (HIVE-21572) HiveRemoveSqCountCheck rule could be enhanced to capture more patterns

2019-04-03 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-21572:
--

 Summary: HiveRemoveSqCountCheck rule could be enhanced to capture 
more patterns 
 Key: HIVE-21572
 URL: https://issues.apache.org/jira/browse/HIVE-21572
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 4.0.0
Reporter: Vineet Garg
Assignee: Vineet Garg
 Attachments: HIVE-21572.1.patch





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70372: HIVE-21427: Syslog storage handler

2019-04-03 Thread j . prasanth . j 


> On April 3, 2019, 9:54 p.m., Jason Dere wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
> > Line 615 (original), 615 (patched)
> > 
> >
> > Just curious about this one, was there a difference between 
> > rbCtx.getRowColumnTypeInfos() and rbCtx.getDataColumnCount()? Or just the 
> > fact that rbCtx.getDataColumnCount() directly returns an int value?

Yes. There is place in encoded reader creation that counts just the non-null 
columns in vector batch. But in here we include all columns including the nulls 
(partition columns at the last) so the text llap reader was throwing exception.


> On April 3, 2019, 9:54 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java
> > Lines 57 (patched)
> > 
> >
> > Is the list of columns from SyslogSerDe fixed to  (facility, severity, 
> > version, ts, hostname, app_name, proc_id, msg_id, structured_data, msg, 
> > unmatched)? If so then should the column list/types be hardcoded rather 
> > than set via LIST_COLUMNS/LIST_COLUMNS_TYPES properties?

Yes. It is fixed schema. make sense.. The columns and types doesn't change 
anyway. Will fix it in next patch.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/#review214337
---


On April 2, 2019, 10:29 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70372/
> ---
> 
> (Updated April 2, 2019, 10:29 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Bugs: HIVE-21427
> https://issues.apache.org/jira/browse/HIVE-21427
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21427: Syslog storage handler
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 777f8b51215523fca8e396ddf77139420666311a 
>   data/files/syslog-hs2-2.log PRE-CREATION 
>   data/files/syslog-hs2.log PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> 96dfbc4b56b6eb3dff6b8e1e42a2371d090426e7 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  9ef7af4eb0c9787a33d2aa4c9a4528b8f356106b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
> 27fe828b7531584138cd002956a9fcc20f238f71 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out 
> PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java
>  eafe0c6d46d448bce287e61fabac0384b12b9295 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java
>  6282078411c4c728beed8e957aa857ed3c02133c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70372/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

[jira] [Created] (HIVE-21571) SHOW COMPACTIONS shows column names as its first output row

2019-04-03 Thread Todd Lipcon (JIRA) 
Todd Lipcon created HIVE-21571:
--

 Summary: SHOW COMPACTIONS shows column names as its first output 
row
 Key: HIVE-21571
 URL: https://issues.apache.org/jira/browse/HIVE-21571
 Project: Hive
  Issue Type: Bug
Reporter: Todd Lipcon


SHOW COMPACTIONS yields a resultset with nice column names, and then the first 
row of data is a repetition of those column names. This is somewhat confusing 
and hard to read.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70372: HIVE-21427: Syslog storage handler

2019-04-03 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/#review214337
---




llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
Line 615 (original), 615 (patched)


Just curious about this one, was there a difference between 
rbCtx.getRowColumnTypeInfos() and rbCtx.getDataColumnCount()? Or just the fact 
that rbCtx.getDataColumnCount() directly returns an int value?



ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java
Lines 57 (patched)


Is the list of columns from SyslogSerDe fixed to  (facility, severity, 
version, ts, hostname, app_name, proc_id, msg_id, structured_data, msg, 
unmatched)? If so then should the column list/types be hardcoded rather than 
set via LIST_COLUMNS/LIST_COLUMNS_TYPES properties?


- Jason Dere


On April 2, 2019, 10:29 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70372/
> ---
> 
> (Updated April 2, 2019, 10:29 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jason Dere.
> 
> 
> Bugs: HIVE-21427
> https://issues.apache.org/jira/browse/HIVE-21427
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21427: Syslog storage handler
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 777f8b51215523fca8e396ddf77139420666311a 
>   data/files/syslog-hs2-2.log PRE-CREATION 
>   data/files/syslog-hs2.log PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> 96dfbc4b56b6eb3dff6b8e1e42a2371d090426e7 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
>  9ef7af4eb0c9787a33d2aa4c9a4528b8f356106b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
> 27fe828b7531584138cd002956a9fcc20f238f71 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out 
> PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java
>  eafe0c6d46d448bce287e61fabac0384b12b9295 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java
>  6282078411c4c728beed8e957aa857ed3c02133c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70372/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

[jira] [Created] (HIVE-21570) Convert llap iomem servlets output to json format

2019-04-03 Thread Antal Sinkovits (JIRA) 
Antal Sinkovits created HIVE-21570:
--

 Summary: Convert llap iomem servlets output to json format
 Key: HIVE-21570
 URL: https://issues.apache.org/jira/browse/HIVE-21570
 Project: Hive
  Issue Type: Improvement
Affects Versions: 4.0.0
Reporter: Antal Sinkovits
Assignee: Antal Sinkovits






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21569) Preparation to bump guava version

2019-04-03 Thread Laszlo Pinter (JIRA) 
Laszlo Pinter created HIVE-21569:


 Summary: Preparation to bump guava version
 Key: HIVE-21569
 URL: https://issues.apache.org/jira/browse/HIVE-21569
 Project: Hive
  Issue Type: Task
  Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter
 Fix For: 4.0.0


This is a preparation work in hive, before updating guava version to 27.0-jre.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21568) HiveRelOptUtil.isRowFilteringPlan should skip Project

2019-04-03 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-21568:
--

 Summary: HiveRelOptUtil.isRowFilteringPlan should skip Project
 Key: HIVE-21568
 URL: https://issues.apache.org/jira/browse/HIVE-21568
 Project: Hive
  Issue Type: Bug
  Components: CBO
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Project operator should not return true in any case, this may trigger 
additional rewritings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21567) Break up DDLTask - extract Function related operations

2019-04-03 Thread Miklos Gergely (JIRA) 
Miklos Gergely created HIVE-21567:
-

 Summary: Break up DDLTask - extract Function related operations
 Key: HIVE-21567
 URL: https://issues.apache.org/jira/browse/HIVE-21567
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 3.1.1
Reporter: Miklos Gergely
Assignee: Miklos Gergely
 Fix For: 4.0.0


DDLTask is a huge class, more than 5000 lines long. The related DDLWork is also 
a huge class, which has a field for each DDL operation it supports. The goal is 
to refactor these in order to have everything cut into more handleable classes 
under the package  org.apache.hadoop.hive.ql.exec.ddl:
 * have a separate class for each operation
 * have a package for each operation group (database ddl, table ddl, etc), so 
the amount of classes under a package is more manageable
 * make all the requests (DDLDesc subclasses) immutable
 * DDLTask should be agnostic to the actual operations
 * right now let's ignore the issue of having some operations handled by 
DDLTask which are not actual DDL operations (lock, unlock, desc...)

In the interim time when there are two DDLTask and DDLWork classes in the code 
base the new ones in the new package are called DDLTask2 and DDLWork2 thus 
avoiding the usage of fully qualified class names where both the old and the 
new classes are in use.

Step #1: extract all the database related operations from the old DDLTask, and 
move them under the new package. Also create the new internal framework.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21566) Support locking during ACID table replication

2019-04-03 Thread mahesh kumar behera (JIRA) 
mahesh kumar behera created HIVE-21566:
--

 Summary: Support locking during ACID table replication  
 Key: HIVE-21566
 URL: https://issues.apache.org/jira/browse/HIVE-21566
 Project: Hive
  Issue Type: Bug
Reporter: mahesh kumar behera
Assignee: mahesh kumar behera


During load of ACID table we need to take lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)