[jira] [Updated] (HIVE-15627) Make hive.vectorized.adaptor.usage.mode=all vectorize all UDFs not just those in supportedGenericUDFs
[ https://issues.apache.org/jira/browse/HIVE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15627: Attachment: HIVE-15627.02.patch > Make hive.vectorized.adaptor.usage.mode=all vectorize all UDFs not just those > in supportedGenericUDFs > - > > Key: HIVE-15627 > URL: https://issues.apache.org/jira/browse/HIVE-15627 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15627.01.patch, HIVE-15627.02.patch > > > Missed this when doing HIVE-14336. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15588) Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc to prevent wrong reuse
[ https://issues.apache.org/jira/browse/HIVE-15588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822739#comment-15822739 ] Matt McCline commented on HIVE-15588: - Patch #3 cratered in 27 sec due to infrastructure issue. > Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc > to prevent wrong reuse > --- > > Key: HIVE-15588 > URL: https://issues.apache.org/jira/browse/HIVE-15588 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15588.01.patch, HIVE-15588.02.patch, > HIVE-15588.03.patch > > > Make sure we don't deallocate a scratch column too quickly and cause result > corruption due to scratch column reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15588) Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc to prevent wrong reuse
[ https://issues.apache.org/jira/browse/HIVE-15588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15588: Status: Patch Available (was: In Progress) > Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc > to prevent wrong reuse > --- > > Key: HIVE-15588 > URL: https://issues.apache.org/jira/browse/HIVE-15588 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15588.01.patch, HIVE-15588.02.patch, > HIVE-15588.03.patch > > > Make sure we don't deallocate a scratch column too quickly and cause result > corruption due to scratch column reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15588) Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc to prevent wrong reuse
[ https://issues.apache.org/jira/browse/HIVE-15588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15588: Attachment: HIVE-15588.03.patch > Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc > to prevent wrong reuse > --- > > Key: HIVE-15588 > URL: https://issues.apache.org/jira/browse/HIVE-15588 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15588.01.patch, HIVE-15588.02.patch, > HIVE-15588.03.patch > > > Make sure we don't deallocate a scratch column too quickly and cause result > corruption due to scratch column reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15505) Include REPLACE, NEXT_DAY, REVERSE to the set of UDFs being Vectorized
[
https://issues.apache.org/jira/browse/HIVE-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822721#comment-15822721
]
Matt McCline edited comment on HIVE-15505 at 1/14/17 7:18 AM:
--
HIVE-15627
was (Author: mmccline):
HIVE-15588
> Include REPLACE, NEXT_DAY, REVERSE to the set of UDFs being Vectorized
> --
>
> Key: HIVE-15505
> URL: https://issues.apache.org/jira/browse/HIVE-15505
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Minor
>
> Currently it does not vectorize and ends up throwing the following msgs
> {noformat}
> physical.Vectorizer: Cannot vectorize UDF GenericUDFNextDay(Column[d_date],
> Const string WE)
> physical.Vectorizer: Cannot vectorize UDF GenericUDFBridge ==> replace
> (Column[d_date_id], Const string a, Const string b)..
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15501) Add INSTR and MONTHS_BETWEEN to UDFs that are Vectorized
[ https://issues.apache.org/jira/browse/HIVE-15501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822725#comment-15822725 ] Matt McCline edited comment on HIVE-15501 at 1/14/17 7:19 AM: -- HIVE-15627 was (Author: mmccline): HIVE-15588 > Add INSTR and MONTHS_BETWEEN to UDFs that are Vectorized > > > Key: HIVE-15501 > URL: https://issues.apache.org/jira/browse/HIVE-15501 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15501.01.patch, HIVE-15501.02.patch > > > Add INSTR to special list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15629) Set DDLTask’s exception with its subtask’s exception
[ https://issues.apache.org/jira/browse/HIVE-15629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-15629: - Status: Patch Available (was: Open) > Set DDLTask’s exception with its subtask’s exception > > > Key: HIVE-15629 > URL: https://issues.apache.org/jira/browse/HIVE-15629 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-15629.000.patch > > > Set DDLTask’s exception with its subtask’s exception, So the exception from > subtask in DDLTask can be propagated to TaskRunner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15629) Set DDLTask’s exception with its subtask’s exception
[ https://issues.apache.org/jira/browse/HIVE-15629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-15629: - Description: Set DDLTask’s exception with its subtask’s exception, So the exception from subtask in DDLTask can be propagated to TaskRunner. (was: Set DDLTask’s exception with its subtask’s exception, So the exception from subtask can be propagated to TaskRunner.) > Set DDLTask’s exception with its subtask’s exception > > > Key: HIVE-15629 > URL: https://issues.apache.org/jira/browse/HIVE-15629 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-15629.000.patch > > > Set DDLTask’s exception with its subtask’s exception, So the exception from > subtask in DDLTask can be propagated to TaskRunner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15629) Set DDLTask’s exception with its subtask’s exception
[ https://issues.apache.org/jira/browse/HIVE-15629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-15629: - Attachment: HIVE-15629.000.patch > Set DDLTask’s exception with its subtask’s exception > > > Key: HIVE-15629 > URL: https://issues.apache.org/jira/browse/HIVE-15629 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-15629.000.patch > > > Set DDLTask’s exception with its subtask’s exception, So the exception from > subtask can be propagated to TaskRunner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15501) Add INSTR and MONTHS_BETWEEN to UDFs that are Vectorized
[ https://issues.apache.org/jira/browse/HIVE-15501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15501: Resolution: Duplicate Status: Resolved (was: Patch Available) HIVE-15588 > Add INSTR and MONTHS_BETWEEN to UDFs that are Vectorized > > > Key: HIVE-15501 > URL: https://issues.apache.org/jira/browse/HIVE-15501 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15501.01.patch, HIVE-15501.02.patch > > > Add INSTR to special list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-15505) Include REPLACE, NEXT_DAY, REVERSE to the set of UDFs being Vectorized
[
https://issues.apache.org/jira/browse/HIVE-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline resolved HIVE-15505.
-
Resolution: Fixed
HIVE-15588
> Include REPLACE, NEXT_DAY, REVERSE to the set of UDFs being Vectorized
> --
>
> Key: HIVE-15505
> URL: https://issues.apache.org/jira/browse/HIVE-15505
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Minor
>
> Currently it does not vectorize and ends up throwing the following msgs
> {noformat}
> physical.Vectorizer: Cannot vectorize UDF GenericUDFNextDay(Column[d_date],
> Const string WE)
> physical.Vectorizer: Cannot vectorize UDF GenericUDFBridge ==> replace
> (Column[d_date_id], Const string a, Const string b)..
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Status: Patch Available (was: Open)
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Attachment: HIVE-15160.01.patch
sounds like an easy fix... not sure if it works for all the test cases. cc'ing
[~ashutoshc]
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-15516) Unable to vectorize select statement having case-when with GenericUDFOPGreaterThan expr
[
https://issues.apache.org/jira/browse/HIVE-15516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline resolved HIVE-15516.
-
Resolution: Duplicate
HIVE-15588
> Unable to vectorize select statement having case-when with
> GenericUDFOPGreaterThan expr
> ---
>
> Key: HIVE-15516
> URL: https://issues.apache.org/jira/browse/HIVE-15516
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>
> First query listed below does not get vectorized; Without "case-when"
> statement it gets vectorized.
> {noformat}
> hive> explain select sum(case when ss_quantity > 1 then ss_quantity *
> ss_wholesale_cost else 0 end) from store_sales;
> explain select sum(case when ss_quantity > 1 then ss_quantity *
> ss_wholesale_cost else 0 end) from store_sales
> OK
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> DagId: rbalamohan_20161227045137_c7a736c6-1812-4c8f-974e-7f7fcc7b1513:28
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> DagName:
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: store_sales
> Statistics: Num rows: 28800426268 Data size: 330048503520
> Basic stats: COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: CASE WHEN ((ss_quantity > 1)) THEN
> ((UDFToDouble(ss_quantity) * ss_wholesale_cost)) ELSE (0) END (type: double)
> outputColumnNames: _col0
> Statistics: Num rows: 28800426268 Data size: 330048503520
> Basic stats: COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: sum(_col0)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 8 Basic stats:
> COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 8 Basic stats:
> COMPLETE Column stats: COMPLETE
> value expressions: _col0 (type: double)
> Execution mode: llap
> LLAP IO: all inputs
> Reducer 2
> Execution mode: vectorized, llap
> Reduce Operator Tree:
> Group By Operator
> aggregations: sum(VALUE._col0)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
> Column stats: COMPLETE
> File Output Operator
> compressed: false
> Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE
> Column stats: COMPLETE
> table:
> input format:
> org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
> serde:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
>
>
> 2016-12-27T04:53:20,507 INFO [16185d97-97f4-477e-9436-4d2b98add389 main]
> physical.Vectorizer: MapWork Operator: SEL could not be vectorized.
> 2016-12-27T04:53:20,507 INFO [16185d97-97f4-477e-9436-4d2b98add389 main]
> physical.Vectorizer: Unable to use the VectorUDFAdaptor. Encountered
> unsupported expr desc : GenericUDFOPGreaterThan(Column[ss_quantity], Const
> int 1)
> 2016-12-27T04:53:20,507 INFO [16185d97-97f4-477e-9436-4d2b98add389 main]
> physical.Vectorizer: Cannot vectorize select expression:
> GenericUDFWhen(GenericUDFOPGreaterThan(Column[ss_quantity], Const int 1),
> GenericUDFOPMultiply(GenericUDFBridge ==> UDFToDouble (Column[ss_quantity]),
> Column[ss_wholesale_cost]), Const int 0)
> 2016-12-27T04:53:20,507 INFO [16185d97-97f4-477e-9436-4d2b98add389 main]
> physical.Vectorizer: MapWork Operator: SEL could not be vectorized.
>
>
> hive> explain select sum(ss_quantity * ss_wholesale_cost) from store_sales;
> explain select sum(ss_quantity * ss_wholesale_cost) from store_sales
> OK
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> DagId: rbalamohan_20161227045112_8311df89-31fb-47ee-ad70-f702a85527cc:27
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> DagName:
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: store_sales
> Statistics: Num rows: 28800426268 Data size:
[jira] [Updated] (HIVE-15588) Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc to prevent wrong reuse
[ https://issues.apache.org/jira/browse/HIVE-15588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15588: Status: In Progress (was: Patch Available) > Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc > to prevent wrong reuse > --- > > Key: HIVE-15588 > URL: https://issues.apache.org/jira/browse/HIVE-15588 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15588.01.patch, HIVE-15588.02.patch > > > Make sure we don't deallocate a scratch column too quickly and cause result > corruption due to scratch column reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15587) Using ChangeManager to copy files in ReplCopyTask
[ https://issues.apache.org/jira/browse/HIVE-15587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822716#comment-15822716 ] Daniel Dai commented on HIVE-15587: --- There are 3 changes in the patch: 1. Adding checksum for bootstrap file listing (_files) 2. Dump and load cmroot in DumpMetaData 3. Using CM to retrieve file (try original location, if deleted or changed, try cmroot instead) > Using ChangeManager to copy files in ReplCopyTask > -- > > Key: HIVE-15587 > URL: https://issues.apache.org/jira/browse/HIVE-15587 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15587.1.patch > > > Currently ReplCopyTask copy files directly from source repo. The files in the > source repo may have been dropped or change. We shall use checksum > transferred to ReplCopyTask to verify. If different, retrieve file from > cmroot instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15587) Using ChangeManager to copy files in ReplCopyTask
[ https://issues.apache.org/jira/browse/HIVE-15587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15587: -- Attachment: HIVE-15587.1.patch The patch is on top of HIVE-15478. > Using ChangeManager to copy files in ReplCopyTask > -- > > Key: HIVE-15587 > URL: https://issues.apache.org/jira/browse/HIVE-15587 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15587.1.patch > > > Currently ReplCopyTask copy files directly from source repo. The files in the > source repo may have been dropped or change. We shall use checksum > transferred to ReplCopyTask to verify. If different, retrieve file from > cmroot instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15478) Add file + checksum list for create table/partition during notification creation (whenever relevant)
[ https://issues.apache.org/jira/browse/HIVE-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15478: -- Attachment: (was: HIVE-15478.2.patch) > Add file + checksum list for create table/partition during notification > creation (whenever relevant) > > > Key: HIVE-15478 > URL: https://issues.apache.org/jira/browse/HIVE-15478 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Daniel Dai > Attachments: HIVE-15478.1.patch, HIVE-15478.2.patch > > > Currently, file list is being generated during REPL DUMP which will result in > inconsistent data getting captured. This ticket is used for event dumping. > Bootstrap dump checksum will be in a different Jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15478) Add file + checksum list for create table/partition during notification creation (whenever relevant)
[ https://issues.apache.org/jira/browse/HIVE-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15478: -- Attachment: HIVE-15478.2.patch > Add file + checksum list for create table/partition during notification > creation (whenever relevant) > > > Key: HIVE-15478 > URL: https://issues.apache.org/jira/browse/HIVE-15478 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Daniel Dai > Attachments: HIVE-15478.1.patch, HIVE-15478.2.patch > > > Currently, file list is being generated during REPL DUMP which will result in > inconsistent data getting captured. This ticket is used for event dumping. > Bootstrap dump checksum will be in a different Jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15478) Add file + checksum list for create table/partition during notification creation (whenever relevant)
[ https://issues.apache.org/jira/browse/HIVE-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15478: -- Attachment: HIVE-15478.2.patch Rebase with master. > Add file + checksum list for create table/partition during notification > creation (whenever relevant) > > > Key: HIVE-15478 > URL: https://issues.apache.org/jira/browse/HIVE-15478 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Daniel Dai > Attachments: HIVE-15478.1.patch, HIVE-15478.2.patch > > > Currently, file list is being generated during REPL DUMP which will result in > inconsistent data getting captured. This ticket is used for event dumping. > Bootstrap dump checksum will be in a different Jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13014) RetryingMetaStoreClient is retrying too aggresievley
[ https://issues.apache.org/jira/browse/HIVE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13014: -- Attachment: HIVE-13014.01.patch > RetryingMetaStoreClient is retrying too aggresievley > > > Key: HIVE-13014 > URL: https://issues.apache.org/jira/browse/HIVE-13014 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13014.01.patch > > > Not all metastore operations are idempotent. For example, commit_txn() > consists of > 1. request from client to server > 2. server action > 3. ack to client > If network connection is broken after (or during) 2 but before 3 happens, > RetryingMetastoreClient will retry the operation thus causing an attempt to > commit the same txn twice (sometimes in concurrently) > The 2nd attempt is guaranteed to fail and thus return an error to the caller > (which doesn't know the operation is being retried), while the first attempt > has actually succeeded. Thus the caller thinks commit failed and will likely > attempt to redo the transactions - not what we want in most cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13696) Monitor fair-scheduler.xml and automatically update/validate jobs submitted to fair-scheduler
[
https://issues.apache.org/jira/browse/HIVE-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822669#comment-15822669
]
Yongzhi Chen commented on HIVE-13696:
-
[~sseth], I attached the fix for the regression 12 hours ago, and it passed
review. Once the pre-commit build is run, I can check in the fix.
Please see HIVE-15615 for details.
> Monitor fair-scheduler.xml and automatically update/validate jobs submitted
> to fair-scheduler
> -
>
> Key: HIVE-13696
> URL: https://issues.apache.org/jira/browse/HIVE-13696
> Project: Hive
> Issue Type: Improvement
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
> Fix For: 2.2.0
>
> Attachments: HIVE-13696.01.patch, HIVE-13696.02.patch,
> HIVE-13696.06.patch, HIVE-13696.08.patch, HIVE-13696.11.patch,
> HIVE-13696.13.patch, HIVE-13696.14.patch
>
>
> Ensure that jobs are placed into the correct queue according to
> {{fair-scheduler.xml}}. Jobs should be placed into the correct queue, and
> users should not be able to submit jobs to queues they do not have access to.
> This patch builds on the existing functionality in {{FairSchedulerShim}} to
> route jobs to user-specific queue based on {{fair-scheduler.xml}}
> configuration (leveraging the Yarn {{QueuePlacementPolicy}} class). In
> addition to configuring job routing at session connect (current behavior),
> the routing is validated per submission to yarn (when impersonation is off).
> A {{FileSystemWatcher}} class is included to monitor changes in the
> {{fair-scheduler.xml}} file (so updates are automatically reloaded when the
> file pointed to by {{yarn.scheduler.fair.allocation.file}} is changed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15627) Make hive.vectorized.adaptor.usage.mode=all vectorize all UDFs not just those in supportedGenericUDFs
[ https://issues.apache.org/jira/browse/HIVE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822660#comment-15822660 ] Gopal V commented on HIVE-15627: +1 tests pending. > Make hive.vectorized.adaptor.usage.mode=all vectorize all UDFs not just those > in supportedGenericUDFs > - > > Key: HIVE-15627 > URL: https://issues.apache.org/jira/browse/HIVE-15627 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15627.01.patch > > > Missed this when doing HIVE-14336. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15617) Improve the avg performance for Range based window
[ https://issues.apache.org/jira/browse/HIVE-15617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15617: Status: Patch Available (was: Open) > Improve the avg performance for Range based window > -- > > Key: HIVE-15617 > URL: https://issues.apache.org/jira/browse/HIVE-15617 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Affects Versions: 1.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15617.1.patch > > > Similar to HIVE-15520, we need to improve the performance for avg(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15617) Improve the avg performance for Range based window
[ https://issues.apache.org/jira/browse/HIVE-15617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15617: Status: Open (was: Patch Available) > Improve the avg performance for Range based window > -- > > Key: HIVE-15617 > URL: https://issues.apache.org/jira/browse/HIVE-15617 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Affects Versions: 1.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15617.1.patch > > > Similar to HIVE-15520, we need to improve the performance for avg(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15546) Optimize Utilities.getInputPaths() so each listStatus of a partition is done in parallel
[ https://issues.apache.org/jira/browse/HIVE-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-15546: Summary: Optimize Utilities.getInputPaths() so each listStatus of a partition is done in parallel (was: Optimize Utilities.getInputPaths()) > Optimize Utilities.getInputPaths() so each listStatus of a partition is done > in parallel > > > Key: HIVE-15546 > URL: https://issues.apache.org/jira/browse/HIVE-15546 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-15546.1.patch, HIVE-15546.2.patch, > HIVE-15546.3.patch > > > When running on blobstores (like S3) where metadata operations (like > listStatus) are costly, Utilities.getInputPaths() can add significant > overhead when setting up the input paths for an MR / Spark / Tez job. > The method performs a listStatus on all input paths in order to check if the > path is empty. If the path is empty, a dummy file is created for the given > partition. This is all done sequentially. This can be really slow when there > are a lot of empty partitions. Even when all partitions have input data, this > can take a long time. > We should either: > (1) Just remove the logic to check if each input path is empty, and handle > any edge cases accordingly. > (2) Multi-thread the listStatus calls -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15622) Remove HWI component from Hive
[ https://issues.apache.org/jira/browse/HIVE-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15622: Status: Patch Available (was: Open) > Remove HWI component from Hive > -- > > Key: HIVE-15622 > URL: https://issues.apache.org/jira/browse/HIVE-15622 > Project: Hive > Issue Type: Task > Components: Web UI >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15622.1.patch > > > This component seems to be obsolete, as it didn't get any meaningful update > since 2012. And we don't see people discussing or complaining issues about > this. Moreover, it caused a number of ptest issues which can be avoided. > We should remove this component as a cleanup effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15627) Make hive.vectorized.adaptor.usage.mode=all vectorize all UDFs not just those in supportedGenericUDFs
[ https://issues.apache.org/jira/browse/HIVE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15627: Status: Patch Available (was: Open) > Make hive.vectorized.adaptor.usage.mode=all vectorize all UDFs not just those > in supportedGenericUDFs > - > > Key: HIVE-15627 > URL: https://issues.apache.org/jira/browse/HIVE-15627 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15627.01.patch > > > Missed this when doing HIVE-14336. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15627) Make hive.vectorized.adaptor.usage.mode=all vectorize all UDFs not just those in supportedGenericUDFs
[ https://issues.apache.org/jira/browse/HIVE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15627: Attachment: HIVE-15627.01.patch > Make hive.vectorized.adaptor.usage.mode=all vectorize all UDFs not just those > in supportedGenericUDFs > - > > Key: HIVE-15627 > URL: https://issues.apache.org/jira/browse/HIVE-15627 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15627.01.patch > > > Missed this when doing HIVE-14336. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15626) beeline exits on ctrl-c instead of canceling the query
[ https://issues.apache.org/jira/browse/HIVE-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822562#comment-15822562 ] Vihang Karajgaonkar commented on HIVE-15626: I can take a look at this [~sershe] > beeline exits on ctrl-c instead of canceling the query > -- > > Key: HIVE-15626 > URL: https://issues.apache.org/jira/browse/HIVE-15626 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Sergey Shelukhin > > I am seeing this in 1.2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15623) Use customized version of netty for llap
[ https://issues.apache.org/jira/browse/HIVE-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822553#comment-15822553 ] Wei Zheng commented on HIVE-15623: -- [~sseth] Do you know how to verify the proper netty is used in the llap classpath? > Use customized version of netty for llap > > > Key: HIVE-15623 > URL: https://issues.apache.org/jira/browse/HIVE-15623 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15623.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15588) Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc to prevent wrong reuse
[ https://issues.apache.org/jira/browse/HIVE-15588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15588: Summary: Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc to prevent wrong reuse (was: Vectorization: Defer deallocation of scratch columns in complex VectorExpressions like VectorUDFAdaptor, VectorUDFCoalesce, etc to prevent wrong reuse) > Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc > to prevent wrong reuse > --- > > Key: HIVE-15588 > URL: https://issues.apache.org/jira/browse/HIVE-15588 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15588.01.patch, HIVE-15588.02.patch > > > Make sure we don't deallocate a scratch column too quickly and cause result > corruption due to scratch column reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15626) beeline exits on ctrl-c instead of canceling the query
[ https://issues.apache.org/jira/browse/HIVE-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15626: Affects Version/s: 1.2.1 > beeline exits on ctrl-c instead of canceling the query > -- > > Key: HIVE-15626 > URL: https://issues.apache.org/jira/browse/HIVE-15626 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Sergey Shelukhin > > I am seeing this in 1.2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15626) beeline exits on ctrl-c instead of canceling the query
[ https://issues.apache.org/jira/browse/HIVE-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15626: Description: I am seeing this in 1.2 (was: I am seeing this in 2.1.) > beeline exits on ctrl-c instead of canceling the query > -- > > Key: HIVE-15626 > URL: https://issues.apache.org/jira/browse/HIVE-15626 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Sergey Shelukhin > > I am seeing this in 1.2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15554) Add task information to LLAP AM heartbeat
[ https://issues.apache.org/jira/browse/HIVE-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822468#comment-15822468 ] Sergey Shelukhin commented on HIVE-15554: - Ping? > Add task information to LLAP AM heartbeat > - > > Key: HIVE-15554 > URL: https://issues.apache.org/jira/browse/HIVE-15554 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15554.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15489) Alternatively use table scan stats for HoS
[ https://issues.apache.org/jira/browse/HIVE-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-15489: Attachment: HIVE-15428.2.patch > Alternatively use table scan stats for HoS > -- > > Key: HIVE-15489 > URL: https://issues.apache.org/jira/browse/HIVE-15489 > Project: Hive > Issue Type: Improvement > Components: Spark, Statistics >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-15428.2.patch, HIVE-15489.wip.patch > > > For MapJoin in HoS, we should provide an option to only use stats in the TS > rather than the populated stats in each of the join branch. This could be > pretty conservative but more reliable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15625) escape1 test fails on Mac
[ https://issues.apache.org/jira/browse/HIVE-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822383#comment-15822383 ] Pengcheng Xiong commented on HIVE-15625: [~sershe], i am sorry that i do not get it. you just remove distinct a, b, c.. etc, why there is collision? > escape1 test fails on Mac > - > > Key: HIVE-15625 > URL: https://issues.apache.org/jira/browse/HIVE-15625 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15625.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15625) escape1 test fails on Mac
[ https://issues.apache.org/jira/browse/HIVE-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15625: Status: Patch Available (was: Open) [~ashutoshc] can you take a look? > escape1 test fails on Mac > - > > Key: HIVE-15625 > URL: https://issues.apache.org/jira/browse/HIVE-15625 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15625.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15625) escape1 test fails on Mac
[ https://issues.apache.org/jira/browse/HIVE-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15625: Attachment: HIVE-15625.patch The reason is that HFS is a joke filesystem. The patch removes lower-case items from input to avoid collision in partition directories in this test. > escape1 test fails on Mac > - > > Key: HIVE-15625 > URL: https://issues.apache.org/jira/browse/HIVE-15625 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15625.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15623) Use customized version of netty for llap
[ https://issues.apache.org/jira/browse/HIVE-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822352#comment-15822352 ] Sergey Shelukhin commented on HIVE-15623: - +1 pending tests > Use customized version of netty for llap > > > Key: HIVE-15623 > URL: https://issues.apache.org/jira/browse/HIVE-15623 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15623.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15621) Remove use of JvmPauseMonitor in LLAP
[ https://issues.apache.org/jira/browse/HIVE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822350#comment-15822350 ] Sergey Shelukhin commented on HIVE-15621: - Is there the Hive pause monitor started anywhere in LLAP? Might be good to start it in LlapDaemon, even though we cannot add it to metrics. cc [~sseth] > Remove use of JvmPauseMonitor in LLAP > - > > Key: HIVE-15621 > URL: https://issues.apache.org/jira/browse/HIVE-15621 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15621.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15590) add separate spnego principal config for LLAP Web UI
[ https://issues.apache.org/jira/browse/HIVE-15590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15590: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master > add separate spnego principal config for LLAP Web UI > > > Key: HIVE-15590 > URL: https://issues.apache.org/jira/browse/HIVE-15590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-15590.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15588) Vectorization: Defer deallocation of scratch columns in complex VectorExpressions like VectorUDFAdaptor, VectorUDFCoalesce, etc to prevent wrong reuse
[ https://issues.apache.org/jira/browse/HIVE-15588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822334#comment-15822334 ] Gopal V commented on HIVE-15588: LGTM - +1. > Vectorization: Defer deallocation of scratch columns in complex > VectorExpressions like VectorUDFAdaptor, VectorUDFCoalesce, etc to prevent > wrong reuse > -- > > Key: HIVE-15588 > URL: https://issues.apache.org/jira/browse/HIVE-15588 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15588.01.patch, HIVE-15588.02.patch > > > Make sure we don't deallocate a scratch column too quickly and cause result > corruption due to scratch column reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15624) sometimes (with schema evolution?) padded char values cannot be converted to integers
[
https://issues.apache.org/jira/browse/HIVE-15624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822312#comment-15822312
]
Sergey Shelukhin commented on HIVE-15624:
-
cc [~mmccline] [~ashutoshc]
> sometimes (with schema evolution?) padded char values cannot be converted to
> integers
> -
>
> Key: HIVE-15624
> URL: https://issues.apache.org/jira/browse/HIVE-15624
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> Noticed while fixing HIVE-15558
> Schema evolution non-vectorized tests, or maybe their previous version on
> 2.1, in some instances have nulls in query results for values that are padded
> due to being char, when converting them to integers (e.g. '5' or '-'
> are converted, but '17 ' cannot be converted and becomes null).
> HIVE-15558, while restoring proper padding to char, also made vectorization
> consistent with non-vectorized Hive in making them null, however that is not
> a desirable behavior.
> I've tried it out in simple queries with casts, and it doesn't happen ('17
> ' of type char(5) is cast to int as expected).
> I think we should fix this behavior for both vectorized and non-vectorized
> case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15624) sometimes (with schema evolution?) padded char values cannot be converted to integers
[
https://issues.apache.org/jira/browse/HIVE-15624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15624:
Description:
Noticed while fixing HIVE-15558
Schema evolution non-vectorized tests, or maybe their previous version on 2.1,
in some instances have nulls in query results for values that are padded due to
being char, when converting them to integers (e.g. '5' or '-' are
converted, but '17 ' cannot be converted and becomes null).
HIVE-15558, while restoring proper padding to char, also made vectorization
consistent with non-vectorized Hive in making them null, however that is not a
desirable behavior.
I've tried it out in simple queries with casts, and it doesn't happen ('17 '
of type char(5) is cast to int as expected).
I think we should fix this behavior for both vectorized and non-vectorized case.
was:
Noticed while fixing HIVE-15558
Schema evolution non-vectorized tests, or maybe their previous version on 2.1,
in some instances have nulls in query results for values that are padded due to
being char (e.g. '5' or '-' are converted, but '17 ' cannot be
converted and becomes null).
HIVE-15558, while restoring proper padding to char, also made vectorization
consistent with non-vectorized Hive in making them null, however that is not a
desirable behavior.
I've tried it out in simple queries with casts, and it doesn't happen ('17 '
of type char(5) is cast to int as expected).
I think we should fix this behavior for both vectorized and non-vectorized case.
> sometimes (with schema evolution?) padded char values cannot be converted to
> integers
> -
>
> Key: HIVE-15624
> URL: https://issues.apache.org/jira/browse/HIVE-15624
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> Noticed while fixing HIVE-15558
> Schema evolution non-vectorized tests, or maybe their previous version on
> 2.1, in some instances have nulls in query results for values that are padded
> due to being char, when converting them to integers (e.g. '5' or '-'
> are converted, but '17 ' cannot be converted and becomes null).
> HIVE-15558, while restoring proper padding to char, also made vectorization
> consistent with non-vectorized Hive in making them null, however that is not
> a desirable behavior.
> I've tried it out in simple queries with casts, and it doesn't happen ('17
> ' of type char(5) is cast to int as expected).
> I think we should fix this behavior for both vectorized and non-vectorized
> case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15534) Update db/table repl.last.id at the end of REPL LOAD of a batch of events
[
https://issues.apache.org/jira/browse/HIVE-15534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-15534:
Status: Patch Available (was: Open)
> Update db/table repl.last.id at the end of REPL LOAD of a batch of events
> -
>
> Key: HIVE-15534
> URL: https://issues.apache.org/jira/browse/HIVE-15534
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-15534.patch
>
>
> Tracking TODO task in ReplSemanticAnalyzer :
> {noformat}
> // TODO : Over here, we need to track a
> Map for every db updated
> // and update repl.last.id for each, if this is a wh-level load, and
> if it is a db-level load,
> // then a single repl.last.id update, and if this is a tbl-lvl load
> which does not alter the
> // table itself, we'll need to update repl.last.id for that as well.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15469) Fix REPL DUMP/LOAD DROP_PTN so it works on non-string-ptn-key tables
[ https://issues.apache.org/jira/browse/HIVE-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-15469: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master. Thanks, [~vgumashta]! > Fix REPL DUMP/LOAD DROP_PTN so it works on non-string-ptn-key tables > > > Key: HIVE-15469 > URL: https://issues.apache.org/jira/browse/HIVE-15469 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Vaibhav Gumashta > Fix For: 2.2.0 > > Attachments: HIVE-15469.1.patch, HIVE-15469.2.patch > > > The current implementation of REPL DUMP/REPL LOAD for DROP_PTN is limited to > dropping partitions whose key types are strings. This needs the tableObj to > be available in the DropPartitionMessage before it can be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13696) Monitor fair-scheduler.xml and automatically update/validate jobs submitted to fair-scheduler
[
https://issues.apache.org/jira/browse/HIVE-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822285#comment-15822285
]
Siddharth Seth commented on HIVE-13696:
---
Can we revert the patch for now, since it is causing failures.
Also, monitoring the fair scheduler xml file will not work on most systems,
since this file is only needed on the YARN ResourceManager page. If HiveServer2
or the hive client is running on any other node - this file will not be
available, and the feature would be broken.
If this is really required, a connection to YARN, along with calls to it's
Queue APIs would be a lot more robust (I'm not absolutely sure on the status of
these APIs at the moment, but a question to yarn-users would solve that).
> Monitor fair-scheduler.xml and automatically update/validate jobs submitted
> to fair-scheduler
> -
>
> Key: HIVE-13696
> URL: https://issues.apache.org/jira/browse/HIVE-13696
> Project: Hive
> Issue Type: Improvement
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
> Fix For: 2.2.0
>
> Attachments: HIVE-13696.01.patch, HIVE-13696.02.patch,
> HIVE-13696.06.patch, HIVE-13696.08.patch, HIVE-13696.11.patch,
> HIVE-13696.13.patch, HIVE-13696.14.patch
>
>
> Ensure that jobs are placed into the correct queue according to
> {{fair-scheduler.xml}}. Jobs should be placed into the correct queue, and
> users should not be able to submit jobs to queues they do not have access to.
> This patch builds on the existing functionality in {{FairSchedulerShim}} to
> route jobs to user-specific queue based on {{fair-scheduler.xml}}
> configuration (leveraging the Yarn {{QueuePlacementPolicy}} class). In
> addition to configuring job routing at session connect (current behavior),
> the routing is validated per submission to yarn (when impersonation is off).
> A {{FileSystemWatcher}} class is included to monitor changes in the
> {{fair-scheduler.xml}} file (so updates are automatically reloaded when the
> file pointed to by {{yarn.scheduler.fair.allocation.file}} is changed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15469) Fix REPL DUMP/LOAD DROP_PTN so it works on non-string-ptn-key tables
[ https://issues.apache.org/jira/browse/HIVE-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822269#comment-15822269 ] Sushanth Sowmyan commented on HIVE-15469: - None of the test failures reported here are related, and since my changes to Vaibhav's patch were trivial and test-related, I'm going to go ahead and check this in with my +1. > Fix REPL DUMP/LOAD DROP_PTN so it works on non-string-ptn-key tables > > > Key: HIVE-15469 > URL: https://issues.apache.org/jira/browse/HIVE-15469 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Vaibhav Gumashta > Attachments: HIVE-15469.1.patch, HIVE-15469.2.patch > > > The current implementation of REPL DUMP/REPL LOAD for DROP_PTN is limited to > dropping partitions whose key types are strings. This needs the tableObj to > be available in the DropPartitionMessage before it can be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-15532) Refactor/cleanup TestReplicationScenario
[ https://issues.apache.org/jira/browse/HIVE-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan resolved HIVE-15532. - Resolution: Duplicate Fix Version/s: 2.2.0 Is incorporated by HIVE-15365 > Refactor/cleanup TestReplicationScenario > > > Key: HIVE-15532 > URL: https://issues.apache.org/jira/browse/HIVE-15532 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Fix For: 2.2.0 > > > TestReplicationScenarios could use a bit of cleanup, based on comments from > reviews: > a) Separate "setup" phase of each test, so that we don't run unnecessary > verifications which aren't testing replication itself, but are verifying that > the env is set up correctly to then test replication. This can be flag-gated > so as to allow it to be turned on at test-dev time, and off during > build/commit unit test time. > b) Better comments inside the tests for what is being set up / tested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-15354) Investigate TestHCatClientNotification#createTable test failure
[
https://issues.apache.org/jira/browse/HIVE-15354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan resolved HIVE-15354.
-
Resolution: Duplicate
Fix Version/s: 2.2.0
Fixed as part of HIVE-15365
> Investigate TestHCatClientNotification#createTable test failure
> ---
>
> Key: HIVE-15354
> URL: https://issues.apache.org/jira/browse/HIVE-15354
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Vaibhav Gumashta
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Fix For: 2.2.0
>
>
> From the test logs, the createTime field for the table object when it was
> stored in notification is 1480852597:
> {code}
> createTime:1480852597, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:onecol, type:string, comment:)],
> location:pfile:/home/hiveptest/104.154.199.46-hiveptest-1/apache-github-source-source/itests/hcatalog-unit/target/warehouse/hcatcreatetable,
> inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=1}), bucketCols:null, sortCols:null,
> parameters:null), partitionKeys:[],
> parameters:{transient_lastDdlTime=1480852596}, viewOriginalText:null,
> viewExpandedText:null, tableType:MANAGED_TABLE)>
> {code}
> However, when the notification is read and the table object is reconstructed
> on client side, the createTime is 1480852596:
> {code}
> createTime:1480852596, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:onecol, type:string, comment:)],
> location:pfile:/home/hiveptest/104.154.199.46-hiveptest-1/apache-github-source-source/itests/hcatalog-unit/target/warehouse/hcatcreatetable,
> inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=1}), bucketCols:null, sortCols:null,
> parameters:null), partitionKeys:[],
> parameters:{transient_lastDdlTime=1480852596}, viewOriginalText:null,
> viewExpandedText:null, tableType:MANAGED_TABLE)>
> {code}
> This also happens intermittently. Need to investigate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-15354) Investigate TestHCatClientNotification#createTable test failure
[
https://issues.apache.org/jira/browse/HIVE-15354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan reassigned HIVE-15354:
---
Assignee: Sushanth Sowmyan (was: Vaibhav Gumashta)
> Investigate TestHCatClientNotification#createTable test failure
> ---
>
> Key: HIVE-15354
> URL: https://issues.apache.org/jira/browse/HIVE-15354
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Vaibhav Gumashta
>Assignee: Sushanth Sowmyan
>Priority: Critical
>
> From the test logs, the createTime field for the table object when it was
> stored in notification is 1480852597:
> {code}
> createTime:1480852597, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:onecol, type:string, comment:)],
> location:pfile:/home/hiveptest/104.154.199.46-hiveptest-1/apache-github-source-source/itests/hcatalog-unit/target/warehouse/hcatcreatetable,
> inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=1}), bucketCols:null, sortCols:null,
> parameters:null), partitionKeys:[],
> parameters:{transient_lastDdlTime=1480852596}, viewOriginalText:null,
> viewExpandedText:null, tableType:MANAGED_TABLE)>
> {code}
> However, when the notification is read and the table object is reconstructed
> on client side, the createTime is 1480852596:
> {code}
> createTime:1480852596, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:onecol, type:string, comment:)],
> location:pfile:/home/hiveptest/104.154.199.46-hiveptest-1/apache-github-source-source/itests/hcatalog-unit/target/warehouse/hcatcreatetable,
> inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=1}), bucketCols:null, sortCols:null,
> parameters:null), partitionKeys:[],
> parameters:{transient_lastDdlTime=1480852596}, viewOriginalText:null,
> viewExpandedText:null, tableType:MANAGED_TABLE)>
> {code}
> This also happens intermittently. Need to investigate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15534) Update db/table repl.last.id at the end of REPL LOAD of a batch of events
[
https://issues.apache.org/jira/browse/HIVE-15534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-15534:
Attachment: HIVE-15534.patch
Patch attached.
> Update db/table repl.last.id at the end of REPL LOAD of a batch of events
> -
>
> Key: HIVE-15534
> URL: https://issues.apache.org/jira/browse/HIVE-15534
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-15534.patch
>
>
> Tracking TODO task in ReplSemanticAnalyzer :
> {noformat}
> // TODO : Over here, we need to track a
> Map for every db updated
> // and update repl.last.id for each, if this is a wh-level load, and
> if it is a db-level load,
> // then a single repl.last.id update, and if this is a tbl-lvl load
> which does not alter the
> // table itself, we'll need to update repl.last.id for that as well.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15534) Update db/table repl.last.id at the end of REPL LOAD of a batch of events
[
https://issues.apache.org/jira/browse/HIVE-15534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822258#comment-15822258
]
Sushanth Sowmyan commented on HIVE-15534:
-
The actual semantic needing implementation is a bit more involved than the one
in the description, The actual semantic implemented is as per the following
comment in code:
{code}
// Now, we need to update repl.last.id for the various parent objects
that were updated.
// This update logic will work differently based on what "level" REPL
LOAD was run on.
// a) If this was a REPL LOAD at a table level, i.e. both
dbNameOrPattern and
// tblNameOrPattern were specified, then the table is the only
thing we should
// update the repl.last.id for.
// b) If this was a db-level REPL LOAD, then we should update the db,
as well as any
// tables affected by partition level operations. (any table level
ops will
// automatically be updated as the table gets updated. Note -
renames will need
// careful handling.
// c) If this was a wh-level REPL LOAD, then we should update every db
for which there
// were events occurring, as well as tables for which there were
ptn-level ops
// happened. Again, renames must be taken care of.
//
// So, what we're going to do is have each event load update dbsUpdated
and tablesUpdated
// accordingly, but ignore updates to tablesUpdated & dbsUpdated in the
case of a
// table-level REPL LOAD, using only the table itself. In the case of a
db-level REPL
// LOAD, we ignore dbsUpdated, but inject our own, and do not ignore
tblsUpdated.
// And for wh-level, we do no special processing, and use all of
dbsUpdated and
// tblsUpdated as-is.
{code}
> Update db/table repl.last.id at the end of REPL LOAD of a batch of events
> -
>
> Key: HIVE-15534
> URL: https://issues.apache.org/jira/browse/HIVE-15534
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-15534.patch
>
>
> Tracking TODO task in ReplSemanticAnalyzer :
> {noformat}
> // TODO : Over here, we need to track a
> Map for every db updated
> // and update repl.last.id for each, if this is a wh-level load, and
> if it is a db-level load,
> // then a single repl.last.id update, and if this is a tbl-lvl load
> which does not alter the
> // table itself, we'll need to update repl.last.id for that as well.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15324) Enable round() function to accept scale argument as non-constants
[
https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822231#comment-15822231
]
Xuefu Zhang commented on HIVE-15324:
Result type needs to determined statically. Scale impacts the returned type, so
different scales for different rows will make result type undeterministic.
> Enable round() function to accept scale argument as non-constants
> -
>
> Key: HIVE-15324
> URL: https://issues.apache.org/jira/browse/HIVE-15324
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15324.1.patch, HIVE-15324.2.patch, HIVE-15324.patch
>
>
> round() function should accept scale argument as non-constants, it will
> enable queries like:
> {quote}
> create table sampletable(c double, d int);
> select round(c,d) from sampletable;
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15431) Round(1234567891.1234567891,50) returns null, result is not consistent with Mysql.
[
https://issues.apache.org/jira/browse/HIVE-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1580#comment-1580
]
Xuefu Zhang commented on HIVE-15431:
We cannot be completely be consistent with mysql. For example, mysql has a max
precision of 68, while we only have 38. Thus, some valid decimal numbers in
mysql would become NULL in Hive.
> Round(1234567891.1234567891,50) returns null, result is not consistent with
> Mysql.
> --
>
> Key: HIVE-15431
> URL: https://issues.apache.org/jira/browse/HIVE-15431
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15431.patch
>
>
> round(1234567891.1234567891,50) returns null. Result is not consistent with
> Mysql.
> Mysql output:
> {quote}
> select round(1234567891.1234567891,50);
> '1234567891.1234567891'
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15616) Improve contents of qfile test output
[ https://issues.apache.org/jira/browse/HIVE-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822210#comment-15822210 ] Thomas Poepping commented on HIVE-15616: Can you open a reviews.apache.org submission for this change? > Improve contents of qfile test output > - > > Key: HIVE-15616 > URL: https://issues.apache.org/jira/browse/HIVE-15616 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 2.1.1 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-15616.patch > > > The current output of the failed qtests has a less than ideal signal to noise > ratio. > We have duplicated stack traces and messages between the error message/stack > trace/error out. > For diff errors the actual difference is missing from the error message and > can be found only in the standard out. > I would like to simplify this output by removing duplications, moving > relevant information to the top. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14706) Lineage information not set properly
[
https://issues.apache.org/jira/browse/HIVE-14706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14706:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
pushed to master. thanks [~ashutoshc] for the review.
> Lineage information not set properly
>
>
> Key: HIVE-14706
> URL: https://issues.apache.org/jira/browse/HIVE-14706
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.1.1
>Reporter: Vimal Sharma
>Assignee: Pengcheng Xiong
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-14706.01.patch
>
>
> I am trying to fetch column level lineage after a CTAS query in a Post
> Execution hook in Hive. Below are the queries:
> {code}
> create table t1(id int, name string);
> create table t2 as select * from t1;
> {code}
> The lineage information is retrieved using the following sample piece of code:
> {code}
> lInfo = hookContext.getLinfo()
> for(Map.Entry e :
> lInfo.entrySet()) {
> System.out.println("Col Lineage Key : " + e.getKey());
> System.out.println("Col Lineage Value: " + e.getValue());
> }
> {code}
> The Dependency field(i.e Col Lineage Value) is coming in as null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14706) Lineage information not set properly
[
https://issues.apache.org/jira/browse/HIVE-14706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14706:
---
Fix Version/s: 2.2.0
> Lineage information not set properly
>
>
> Key: HIVE-14706
> URL: https://issues.apache.org/jira/browse/HIVE-14706
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.1.1
>Reporter: Vimal Sharma
>Assignee: Pengcheng Xiong
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-14706.01.patch
>
>
> I am trying to fetch column level lineage after a CTAS query in a Post
> Execution hook in Hive. Below are the queries:
> {code}
> create table t1(id int, name string);
> create table t2 as select * from t1;
> {code}
> The lineage information is retrieved using the following sample piece of code:
> {code}
> lInfo = hookContext.getLinfo()
> for(Map.Entry e :
> lInfo.entrySet()) {
> System.out.println("Col Lineage Key : " + e.getKey());
> System.out.println("Col Lineage Value: " + e.getValue());
> }
> {code}
> The Dependency field(i.e Col Lineage Value) is coming in as null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14706) Lineage information not set properly
[
https://issues.apache.org/jira/browse/HIVE-14706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14706:
---
Affects Version/s: 2.1.0
2.1.1
> Lineage information not set properly
>
>
> Key: HIVE-14706
> URL: https://issues.apache.org/jira/browse/HIVE-14706
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.1.1
>Reporter: Vimal Sharma
>Assignee: Pengcheng Xiong
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-14706.01.patch
>
>
> I am trying to fetch column level lineage after a CTAS query in a Post
> Execution hook in Hive. Below are the queries:
> {code}
> create table t1(id int, name string);
> create table t2 as select * from t1;
> {code}
> The lineage information is retrieved using the following sample piece of code:
> {code}
> lInfo = hookContext.getLinfo()
> for(Map.Entry e :
> lInfo.entrySet()) {
> System.out.println("Col Lineage Key : " + e.getKey());
> System.out.println("Col Lineage Value: " + e.getValue());
> }
> {code}
> The Dependency field(i.e Col Lineage Value) is coming in as null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15623) Use customized version of netty for llap
[ https://issues.apache.org/jira/browse/HIVE-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822197#comment-15822197 ] Wei Zheng commented on HIVE-15623: -- [~sershe] Can you please review? > Use customized version of netty for llap > > > Key: HIVE-15623 > URL: https://issues.apache.org/jira/browse/HIVE-15623 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15623.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15623) Use customized version of netty for llap
[ https://issues.apache.org/jira/browse/HIVE-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15623: - Attachment: HIVE-15623.1.patch > Use customized version of netty for llap > > > Key: HIVE-15623 > URL: https://issues.apache.org/jira/browse/HIVE-15623 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15623.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15622) Remove HWI component from Hive
[ https://issues.apache.org/jira/browse/HIVE-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822153#comment-15822153 ] Wei Zheng commented on HIVE-15622: -- [~ashutoshc] Can you please review? > Remove HWI component from Hive > -- > > Key: HIVE-15622 > URL: https://issues.apache.org/jira/browse/HIVE-15622 > Project: Hive > Issue Type: Task > Components: Web UI >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15622.1.patch > > > This component seems to be obsolete, as it didn't get any meaningful update > since 2012. And we don't see people discussing or complaining issues about > this. Moreover, it caused a number of ptest issues which can be avoided. > We should remove this component as a cleanup effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15569) failures in RetryingHMSHandler. do not get retried
[ https://issues.apache.org/jira/browse/HIVE-15569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822148#comment-15822148 ] Vihang Karajgaonkar commented on HIVE-15569: Hi [~spena] and [~thejas] Can you please review the patch? The test failures are unrelated and have been failing recently for other patches too. > failures in RetryingHMSHandler. do not get retried > > > Key: HIVE-15569 > URL: https://issues.apache.org/jira/browse/HIVE-15569 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Vihang Karajgaonkar > Attachments: HIVE-15569.01.patch, HIVE-15569.02.patch > > > RetryingHMSHandler. is called during Hive metastore startup, and any > transient db failures during that call are not retried. This can result in > failure for HiveMetastore startup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15622) Remove HWI component from Hive
[ https://issues.apache.org/jira/browse/HIVE-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15622: - Attachment: HIVE-15622.1.patch > Remove HWI component from Hive > -- > > Key: HIVE-15622 > URL: https://issues.apache.org/jira/browse/HIVE-15622 > Project: Hive > Issue Type: Task > Components: Web UI >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15622.1.patch > > > This component seems to be obsolete, as it didn't get any meaningful update > since 2012. And we don't see people discussing or complaining issues about > this. Moreover, it caused a number of ptest issues which can be avoided. > We should remove this component as a cleanup effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15617) Improve the avg performance for Range based window
[ https://issues.apache.org/jira/browse/HIVE-15617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822141#comment-15822141 ] Aihua Xu commented on HIVE-15617: - patch-1: avg() will first call sum() to get the sum and then calculate the average from sum/num_of_rows. > Improve the avg performance for Range based window > -- > > Key: HIVE-15617 > URL: https://issues.apache.org/jira/browse/HIVE-15617 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Affects Versions: 1.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15617.1.patch > > > Similar to HIVE-15520, we need to improve the performance for avg(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Status: Patch Available (was: Open)
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch, HIVE-15472.2.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Attachment: HIVE-15472.2.patch
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch, HIVE-15472.2.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Status: Open (was: Patch Available)
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch, HIVE-15472.2.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822134#comment-15822134
]
Tao Li commented on HIVE-15472:
---
According to the error below, the log4j-provider.properties files shaded to the
jdbc jar is specifying the classes from "org.apache.logging" which have been
renamed in HIVE-14992. Apparently the relocation/renaming does not update the
properties file. So for now the quick fix is to revert relocation for
"org.apache.logging". Will upload another patch for that.
ERROR StatusLogger Unable to create class
org.apache.logging.log4j.core.impl.Log4jContextFactory specified in
jar:file:/Users/tli/dev/hive/jdbc/target/hive-jdbc-2.2.0-SNAPSHOT-standalone.jar!/META-INF/log4j-provider.properties
java.lang.ClassNotFoundException:
org.apache.logging.log4j.core.impl.Log4jContextFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15621) Remove use of JvmPauseMonitor in LLAP
[ https://issues.apache.org/jira/browse/HIVE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15621: - Status: Patch Available (was: Open) > Remove use of JvmPauseMonitor in LLAP > - > > Key: HIVE-15621 > URL: https://issues.apache.org/jira/browse/HIVE-15621 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15621.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15621) Remove use of JvmPauseMonitor in LLAP
[ https://issues.apache.org/jira/browse/HIVE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822128#comment-15822128 ] Wei Zheng commented on HIVE-15621: -- [~sershe] Can you please review? > Remove use of JvmPauseMonitor in LLAP > - > > Key: HIVE-15621 > URL: https://issues.apache.org/jira/browse/HIVE-15621 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15621.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15621) Remove use of JvmPauseMonitor in LLAP
[ https://issues.apache.org/jira/browse/HIVE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15621: - Attachment: HIVE-15621.1.patch > Remove use of JvmPauseMonitor in LLAP > - > > Key: HIVE-15621 > URL: https://issues.apache.org/jira/browse/HIVE-15621 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15621.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15623) Use customized version of netty for llap
[ https://issues.apache.org/jira/browse/HIVE-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15623: - Issue Type: Task (was: Bug) > Use customized version of netty for llap > > > Key: HIVE-15623 > URL: https://issues.apache.org/jira/browse/HIVE-15623 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15622) Remove HWI component from Hive
[ https://issues.apache.org/jira/browse/HIVE-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15622: - Description: This component seems to be obsolete, as it didn't get any meaningful update since 2012. And we don't see people discussing or complaining issues about this. Moreover, it caused a number of ptest issues which can be avoided. We should remove this component as a cleanup effort. was: This component seems to be obsolete, as it didn't get any meaningful update since 2012. And we don't see people discussing or complaining issues about this. Moreover, it caused a number of test issues which can be avoided. We should remove this component as a cleanup effort. > Remove HWI component from Hive > -- > > Key: HIVE-15622 > URL: https://issues.apache.org/jira/browse/HIVE-15622 > Project: Hive > Issue Type: Task > Components: Web UI >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > This component seems to be obsolete, as it didn't get any meaningful update > since 2012. And we don't see people discussing or complaining issues about > this. Moreover, it caused a number of ptest issues which can be avoided. > We should remove this component as a cleanup effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15622) Remove HWI component from Hive
[ https://issues.apache.org/jira/browse/HIVE-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15622: - Description: This component seems to be obsolete, as it didn't get any meaningful update since 2012. And we don't see people discussing or complaining issues about this. Moreover, it caused a number of test issues which can be avoided. We should remove this component as a cleanup effort. was: This component seems to be obsolete, as it didn't get any meaningful update since 2013. And we don't see people discussing or complaining issues about this. Moreover, it caused a number of test issues which can be avoided. We should remove this component as a cleanup effort. > Remove HWI component from Hive > -- > > Key: HIVE-15622 > URL: https://issues.apache.org/jira/browse/HIVE-15622 > Project: Hive > Issue Type: Task > Components: Web UI >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > This component seems to be obsolete, as it didn't get any meaningful update > since 2012. And we don't see people discussing or complaining issues about > this. Moreover, it caused a number of test issues which can be avoided. > We should remove this component as a cleanup effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15622) Remove HWI component from Hive
[ https://issues.apache.org/jira/browse/HIVE-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15622: - Description: This component seems to be obsolete, as it didn't get any meaningful update since 2013. And we don't see people discussing or complaining issues about this. Moreover, it caused a number of test issues which can be avoided. We should remove this component as a cleanup effort. was: This component seems to be obsolete, as it didn't get any update since 2013. And we don't see people discussing or complaining issues about this. Moreover, it caused a number of test issues which can be avoided. We should remove this component as a cleanup effort. > Remove HWI component from Hive > -- > > Key: HIVE-15622 > URL: https://issues.apache.org/jira/browse/HIVE-15622 > Project: Hive > Issue Type: Task > Components: Web UI >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > This component seems to be obsolete, as it didn't get any meaningful update > since 2013. And we don't see people discussing or complaining issues about > this. Moreover, it caused a number of test issues which can be avoided. > We should remove this component as a cleanup effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15622) Remove HWI component from Hive
[ https://issues.apache.org/jira/browse/HIVE-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15622: - Description: This component seems to be obsolete, as it didn't get any update since 2013. And we don't see people discussing or complaining issues about this. Moreover, it caused a number of test issues which can be avoided. We should remove this component as a cleanup effort. was:This component seems to be obsolete, as it didn't get any update since 2013. And we don't see people discussing or complaining issues about this. We should remove this component as a cleanup effort. > Remove HWI component from Hive > -- > > Key: HIVE-15622 > URL: https://issues.apache.org/jira/browse/HIVE-15622 > Project: Hive > Issue Type: Task > Components: Web UI >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > This component seems to be obsolete, as it didn't get any update since 2013. > And we don't see people discussing or complaining issues about this. > Moreover, it caused a number of test issues which can be avoided. > We should remove this component as a cleanup effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15622) Remove HWI component from Hive
[ https://issues.apache.org/jira/browse/HIVE-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15622: - Description: This component seems to be obsolete, as it didn't get any update since 2013. And we don't see people discussing or complaining issues about this. We should remove this component as a cleanup effort. > Remove HWI component from Hive > -- > > Key: HIVE-15622 > URL: https://issues.apache.org/jira/browse/HIVE-15622 > Project: Hive > Issue Type: Task > Components: Web UI >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > This component seems to be obsolete, as it didn't get any update since 2013. > And we don't see people discussing or complaining issues about this. We > should remove this component as a cleanup effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15622) Remove HWI component from Hive
[ https://issues.apache.org/jira/browse/HIVE-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15622: - Issue Type: Task (was: Bug) > Remove HWI component from Hive > -- > > Key: HIVE-15622 > URL: https://issues.apache.org/jira/browse/HIVE-15622 > Project: Hive > Issue Type: Task > Components: Web UI >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15621) Remove use of JvmPauseMonitor in LLAP
[ https://issues.apache.org/jira/browse/HIVE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15621: - Issue Type: Task (was: Bug) > Remove use of JvmPauseMonitor in LLAP > - > > Key: HIVE-15621 > URL: https://issues.apache.org/jira/browse/HIVE-15621 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15572) Improve the response time for query canceling when it happens during acquiring locks
[
https://issues.apache.org/jira/browse/HIVE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822088#comment-15822088
]
Chaoyu Tang commented on HIVE-15572:
LGTM, +1
> Improve the response time for query canceling when it happens during
> acquiring locks
>
>
> Key: HIVE-15572
> URL: https://issues.apache.org/jira/browse/HIVE-15572
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-15572.1.patch, HIVE-15572.2.patch
>
>
> When query canceling command sent during Hive Acquire locks (from zookeeper),
> hive will finish acquiring all the locks and release them. As it is shown in
> the following log:
> It took 165 s to finish acquire the lock,then spend 81s to release them.
> We can improve the performance by not acquiring any more locks and releasing
> held locks when the query canceling command is received.
> {noformat}
> Background-Pool: Thread-224]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:50:35,413 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-224]: method=acquireReadWriteLocks from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,671 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: method=acquireReadWriteLocks start=1483469295080 end=1483469460671
> duration=165591 from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,672 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,672 ERROR org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Background-Pool: Thread-218]: FAILED: query select count(*) from
> manyparttbl has been cancelled
> 2017-01-03 10:51:00,673 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:40,755 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-215]: start=1483469419487 end=1483469500755 duration=81268
> from=org.apache.hadoop.hive.ql.Driver>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15572) Improve the response time for query canceling when it happens during acquiring locks
[
https://issues.apache.org/jira/browse/HIVE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-15572:
Attachment: HIVE-15572.2.patch
> Improve the response time for query canceling when it happens during
> acquiring locks
>
>
> Key: HIVE-15572
> URL: https://issues.apache.org/jira/browse/HIVE-15572
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-15572.1.patch, HIVE-15572.2.patch
>
>
> When query canceling command sent during Hive Acquire locks (from zookeeper),
> hive will finish acquiring all the locks and release them. As it is shown in
> the following log:
> It took 165 s to finish acquire the lock,then spend 81s to release them.
> We can improve the performance by not acquiring any more locks and releasing
> held locks when the query canceling command is received.
> {noformat}
> Background-Pool: Thread-224]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:50:35,413 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-224]: method=acquireReadWriteLocks from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,671 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: method=acquireReadWriteLocks start=1483469295080 end=1483469460671
> duration=165591 from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,672 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,672 ERROR org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Background-Pool: Thread-218]: FAILED: query select count(*) from
> manyparttbl has been cancelled
> 2017-01-03 10:51:00,673 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:40,755 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-215]: start=1483469419487 end=1483469500755 duration=81268
> from=org.apache.hadoop.hive.ql.Driver>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15572) Improve the response time for query canceling when it happens during acquiring locks
[
https://issues.apache.org/jira/browse/HIVE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-15572:
Attachment: (was: HIVE-15572.2.patch)
> Improve the response time for query canceling when it happens during
> acquiring locks
>
>
> Key: HIVE-15572
> URL: https://issues.apache.org/jira/browse/HIVE-15572
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-15572.1.patch, HIVE-15572.2.patch
>
>
> When query canceling command sent during Hive Acquire locks (from zookeeper),
> hive will finish acquiring all the locks and release them. As it is shown in
> the following log:
> It took 165 s to finish acquire the lock,then spend 81s to release them.
> We can improve the performance by not acquiring any more locks and releasing
> held locks when the query canceling command is received.
> {noformat}
> Background-Pool: Thread-224]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:50:35,413 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-224]: method=acquireReadWriteLocks from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,671 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: method=acquireReadWriteLocks start=1483469295080 end=1483469460671
> duration=165591 from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,672 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,672 ERROR org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Background-Pool: Thread-218]: FAILED: query select count(*) from
> manyparttbl has been cancelled
> 2017-01-03 10:51:00,673 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:40,755 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-215]: start=1483469419487 end=1483469500755 duration=81268
> from=org.apache.hadoop.hive.ql.Driver>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15537) Nested column pruning: fix issue when selecting struct field from array/map element (part 2)
[
https://issues.apache.org/jira/browse/HIVE-15537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15537:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks [~Ferd] for the review.
> Nested column pruning: fix issue when selecting struct field from array/map
> element (part 2)
>
>
> Key: HIVE-15537
> URL: https://issues.apache.org/jira/browse/HIVE-15537
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.2.0
>
> Attachments: HIVE-15537.1.patch, HIVE-15537.2.patch,
> HIVE-15537.3.patch, HIVE-15537.4.patch, HIVE-15537.5.patch
>
>
> HIVE-15507 only addresses the issue of
> {code}
> SELECT arr[0].f FROM tbl
> {code}
> However, it didn't handle:
> {code}
> SELECT arr[0].f.g FROM tbl
> {code}
> In this case the current code will generate a path {{arr.g}}, which is wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15617) Improve the avg performance for Range based window
[ https://issues.apache.org/jira/browse/HIVE-15617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15617: Status: Patch Available (was: Open) > Improve the avg performance for Range based window > -- > > Key: HIVE-15617 > URL: https://issues.apache.org/jira/browse/HIVE-15617 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Affects Versions: 1.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15617.1.patch > > > Similar to HIVE-15520, we need to improve the performance for avg(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15618) Change hive-blobstore tests to run with Tez by default
[ https://issues.apache.org/jira/browse/HIVE-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Poepping updated HIVE-15618: --- Status: Patch Available (was: Open) > Change hive-blobstore tests to run with Tez by default > -- > > Key: HIVE-15618 > URL: https://issues.apache.org/jira/browse/HIVE-15618 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-15618.patch > > > Ever since the upgrade to Hive 2, Tez has been the default execution engine > for Hive. To match that fact, it makes sense to run our tests against Tez, > rather than MR. This should more fully validate functionality against what we > consider to be Hive defaults. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15583) CTAS query removes leading underscore from column names with CBO
[
https://issues.apache.org/jira/browse/HIVE-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822040#comment-15822040
]
Ashutosh Chauhan commented on HIVE-15583:
-
+1
> CTAS query removes leading underscore from column names with CBO
>
>
> Key: HIVE-15583
> URL: https://issues.apache.org/jira/browse/HIVE-15583
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-15583.01.patch, HIVE-15583.patch
>
>
> L209 in PlanModifierForASTConv.java:
> {code:java}
> if (colAlias.startsWith("_")) {
> colAlias = colAlias.substring(1);
> colAlias = getNewColAlias(newSelAliases, colAlias);
> }
> {code}
> I would like to explore if we can just remove this limitation.
> For instance, due to this issue, when we create a table with Druid storage
> handler, we need to add an additional underscore for column names as Druid is
> expecting columns with a certain name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15618) Change hive-blobstore tests to run with Tez by default
[ https://issues.apache.org/jira/browse/HIVE-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Poepping updated HIVE-15618: --- Attachment: HIVE-15618.patch > Change hive-blobstore tests to run with Tez by default > -- > > Key: HIVE-15618 > URL: https://issues.apache.org/jira/browse/HIVE-15618 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-15618.patch > > > Ever since the upgrade to Hive 2, Tez has been the default execution engine > for Hive. To match that fact, it makes sense to run our tests against Tez, > rather than MR. This should more fully validate functionality against what we > consider to be Hive defaults. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15582) Druid CTAS should support BYTE/SHORT/INT types
[
https://issues.apache.org/jira/browse/HIVE-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822033#comment-15822033
]
Ashutosh Chauhan commented on HIVE-15582:
-
+1 pending tests
> Druid CTAS should support BYTE/SHORT/INT types
> --
>
> Key: HIVE-15582
> URL: https://issues.apache.org/jira/browse/HIVE-15582
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-15582.02.patch, HIVE-15582.patch
>
>
> Currently these types are not recognized and we throw an exception when we
> try to create a table with them.
> {noformat}
> Caused by: org.apache.hadoop.hive.serde2.SerDeException: Unknown type: INT
> at
> org.apache.hadoop.hive.druid.serde.DruidSerDe.serialize(DruidSerDe.java:414)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:715)
> ... 22 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15572) Improve the response time for query canceling when it happens during acquiring locks
[
https://issues.apache.org/jira/browse/HIVE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-15572:
Attachment: HIVE-15572.2.patch
PATCH2 fixes according to the review.
> Improve the response time for query canceling when it happens during
> acquiring locks
>
>
> Key: HIVE-15572
> URL: https://issues.apache.org/jira/browse/HIVE-15572
> Project: Hive
> Issue Type: Improvement
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-15572.1.patch, HIVE-15572.2.patch
>
>
> When query canceling command sent during Hive Acquire locks (from zookeeper),
> hive will finish acquiring all the locks and release them. As it is shown in
> the following log:
> It took 165 s to finish acquire the lock,then spend 81s to release them.
> We can improve the performance by not acquiring any more locks and releasing
> held locks when the query canceling command is received.
> {noformat}
> Background-Pool: Thread-224]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:50:35,413 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-224]: method=acquireReadWriteLocks from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,671 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: method=acquireReadWriteLocks start=1483469295080 end=1483469460671
> duration=165591 from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,672 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:00,672 ERROR org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Background-Pool: Thread-218]: FAILED: query select count(*) from
> manyparttbl has been cancelled
> 2017-01-03 10:51:00,673 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-218]: from=org.apache.hadoop.hive.ql.Driver>
> 2017-01-03 10:51:40,755 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [HiveServer2-Background-Pool: Thread-215]: start=1483469419487 end=1483469500755 duration=81268
> from=org.apache.hadoop.hive.ql.Driver>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14706) Lineage information not set properly
[
https://issues.apache.org/jira/browse/HIVE-14706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821995#comment-15821995
]
Ashutosh Chauhan commented on HIVE-14706:
-
+1
> Lineage information not set properly
>
>
> Key: HIVE-14706
> URL: https://issues.apache.org/jira/browse/HIVE-14706
> Project: Hive
> Issue Type: Bug
>Reporter: Vimal Sharma
>Assignee: Pengcheng Xiong
>Priority: Critical
> Attachments: HIVE-14706.01.patch
>
>
> I am trying to fetch column level lineage after a CTAS query in a Post
> Execution hook in Hive. Below are the queries:
> {code}
> create table t1(id int, name string);
> create table t2 as select * from t1;
> {code}
> The lineage information is retrieved using the following sample piece of code:
> {code}
> lInfo = hookContext.getLinfo()
> for(Map.Entry e :
> lInfo.entrySet()) {
> System.out.println("Col Lineage Key : " + e.getKey());
> System.out.println("Col Lineage Value: " + e.getValue());
> }
> {code}
> The Dependency field(i.e Col Lineage Value) is coming in as null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15617) Improve the avg performance for Range based window
[ https://issues.apache.org/jira/browse/HIVE-15617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15617: Attachment: HIVE-15617.1.patch > Improve the avg performance for Range based window > -- > > Key: HIVE-15617 > URL: https://issues.apache.org/jira/browse/HIVE-15617 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Affects Versions: 1.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15617.1.patch > > > Similar to HIVE-15520, we need to improve the performance for avg(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15617) Improve the avg performance for Range based window
[ https://issues.apache.org/jira/browse/HIVE-15617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15617: Attachment: (was: HIVE-15617.1.patch) > Improve the avg performance for Range based window > -- > > Key: HIVE-15617 > URL: https://issues.apache.org/jira/browse/HIVE-15617 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Affects Versions: 1.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15617.1.patch > > > Similar to HIVE-15520, we need to improve the performance for avg(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15617) Improve the avg performance for Range based window
[ https://issues.apache.org/jira/browse/HIVE-15617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15617: Attachment: HIVE-15617.1.patch > Improve the avg performance for Range based window > -- > > Key: HIVE-15617 > URL: https://issues.apache.org/jira/browse/HIVE-15617 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Affects Versions: 1.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15617.1.patch > > > Similar to HIVE-15520, we need to improve the performance for avg(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15615) Fix unit tests failures caused by HIVE-13696
[
https://issues.apache.org/jira/browse/HIVE-15615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821942#comment-15821942
]
Aihua Xu commented on HIVE-15615:
-
The patch looks good. +1.
Seems we should have single place to have the username. Right now it's in
multiple places.
> Fix unit tests failures caused by HIVE-13696
>
>
> Key: HIVE-15615
> URL: https://issues.apache.org/jira/browse/HIVE-15615
> Project: Hive
> Issue Type: Bug
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-15615.1.patch
>
>
> Following unit tests failed with same stack:
> org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveAuthorizerCheckInvocation
> org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveAuthorizerShowFilters
> {noformat}
> 2017-01-11T15:02:27,774 ERROR [main] ql.Driver: FAILED: NullPointerException
> null
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementRule.cleanName(QueuePlacementRule.java:351)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementRule$User.getQueueForApp(QueuePlacementRule.java:132)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementRule.assignAppToQueue(QueuePlacementRule.java:74)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.assignAppToQueue(QueuePlacementPolicy.java:167)
> at
> org.apache.hadoop.hive.schshim.FairSchedulerShim.setJobQueueForUserInternal(FairSchedulerShim.java:96)
> at
> org.apache.hadoop.hive.schshim.FairSchedulerShim.validateQueueConfiguration(FairSchedulerShim.java:82)
> at
> org.apache.hadoop.hive.ql.session.YarnFairScheduling.validateYarnQueue(YarnFairScheduling.java:68)
> at org.apache.hadoop.hive.ql.Driver.configureScheduling(Driver.java:671)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:543)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1313)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1233)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1223)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15616) Improve contents of qfile test output
[ https://issues.apache.org/jira/browse/HIVE-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821919#comment-15821919 ] Barna Zsombor Klara commented on HIVE-15616: The duplicated issues were created when the ASF Jira was having issues and unfortunately I cannot close/resolve them. > Improve contents of qfile test output > - > > Key: HIVE-15616 > URL: https://issues.apache.org/jira/browse/HIVE-15616 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 2.1.1 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-15616.patch > > > The current output of the failed qtests has a less than ideal signal to noise > ratio. > We have duplicated stack traces and messages between the error message/stack > trace/error out. > For diff errors the actual difference is missing from the error message and > can be found only in the standard out. > I would like to simplify this output by removing duplications, moving > relevant information to the top. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15269) Dynamic Min-Max runtime-filtering for Tez
[
https://issues.apache.org/jira/browse/HIVE-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal updated HIVE-15269:
--
Attachment: HIVE-15269.12.patch
Removed a frivolous logic to remove semi join optimizations around map joins.
Handle cycles due to map side joins and semi join combo.
> Dynamic Min-Max runtime-filtering for Tez
> -
>
> Key: HIVE-15269
> URL: https://issues.apache.org/jira/browse/HIVE-15269
> Project: Hive
> Issue Type: New Feature
>Reporter: Jason Dere
>Assignee: Deepak Jaiswal
> Attachments: HIVE-15269.10.patch, HIVE-15269.11.patch,
> HIVE-15269.12.patch, HIVE-15269.1.patch, HIVE-15269.2.patch,
> HIVE-15269.3.patch, HIVE-15269.4.patch, HIVE-15269.5.patch,
> HIVE-15269.6.patch, HIVE-15269.7.patch, HIVE-15269.8.patch, HIVE-15269.9.patch
>
>
> If a dimension table and fact table are joined:
> {noformat}
> select *
> from store join store_sales on (store.id = store_sales.store_id)
> where store.s_store_name = 'My Store'
> {noformat}
> One optimization that can be done is to get the min/max store id values that
> come out of the scan/filter of the store table, and send this min/max value
> (via Tez edge) to the task which is scanning the store_sales table.
> We can add a BETWEEN(min, max) predicate to the store_sales TableScan, where
> this predicate can be pushed down to the storage handler (for example for ORC
> formats). Pushing a min/max predicate to the ORC reader would allow us to
> avoid having to entire whole row groups during the table scan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15616) Improve contents of qfile test output
[ https://issues.apache.org/jira/browse/HIVE-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-15616: --- Attachment: HIVE-15616.patch First draft of the change. I would like to see how the unit test failures look if I add the diff value to the error message. > Improve contents of qfile test output > - > > Key: HIVE-15616 > URL: https://issues.apache.org/jira/browse/HIVE-15616 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 2.1.1 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-15616.patch > > > The current output of the failed qtests has a less than ideal signal to noise > ratio. > We have duplicated stack traces and messages between the error message/stack > trace/error out. > For diff errors the actual difference is missing from the error message and > can be found only in the standard out. > I would like to simplify this output by removing duplications, moving > relevant information to the top. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15616) Improve contents of qfile test output
[ https://issues.apache.org/jira/browse/HIVE-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-15616: --- Status: Patch Available (was: Open) > Improve contents of qfile test output > - > > Key: HIVE-15616 > URL: https://issues.apache.org/jira/browse/HIVE-15616 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 2.1.1 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-15616.patch > > > The current output of the failed qtests has a less than ideal signal to noise > ratio. > We have duplicated stack traces and messages between the error message/stack > trace/error out. > For diff errors the actual difference is missing from the error message and > can be found only in the standard out. > I would like to simplify this output by removing duplications, moving > relevant information to the top. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
