date:20140422

[jira] [Updated] (HIVE-6955) ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait Propagation in Joins

2014-04-22 Thread Harish Butani (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6955:


Attachment: HIVE-6955.1.patch

> ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait 
> Propagation in Joins
> 
>
> Key: HIVE-6955
> URL: https://issues.apache.org/jira/browse/HIVE-6955
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-6955.1.patch
>
>
> For tpcds Q15:
> {code}
> explain
> select ca_zip, sum(cs_sales_price)
> from catalog_sales, customer, customer_address, date_dim
> where catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
>   and customer.c_current_addr_sk = customer_address.ca_address_sk
>   and (substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
>   '85392', '85460', '80348', '81792')
>or ca_state in ('CA','WA','GA')
>or cs_sales_price > 500)
>   and catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
>   and d_qoy = 2 and d_year = 2001
> group by ca_zip
> order by ca_zip
> limit 100;
> {code}
> The Traits setup for the Operators are:
> {code}
> FIL[23]: bucketCols=[[]],numBuckets=-1
> RS[11]: bucketCols=[[VALUE._col0]],numBuckets=-1
> JOIN[12]: bucketCols=[[_col71], [_col71]],numBuckets=-1
> FIL[13]: bucketCols=[[_col71], [_col71]],numBuckets=-1
> SEL[14]: bucketCols=[[_col71], [_col71]],numBuckets=-1
> GBY[15]: bucketCols=[[_col0]],numBuckets=-1
> RS[16]: bucketCols=[[KEY._col0]],numBuckets=-1
> GBY[17]: bucketCols=[[_col0]],numBuckets=-1
> SEL[18]: bucketCols=[[_col0]],numBuckets=-1
> LIM[21]: bucketCols=[[_col0]],numBuckets=-1
> FS[22]: bucketCols=[[_col0]],numBuckets=-1
> TS[3]: bucketCols=[[]],numBuckets=-1
> RS[5]: bucketCols=[[VALUE._col0]],numBuckets=-1
> JOIN[6]: bucketCols=[[_col3], [_col36]],numBuckets=-1
> RS[7]: bucketCols=[[VALUE._col40]],numBuckets=-1
> JOIN[9]: bucketCols=[[_col40], [_col0]],numBuckets=-1
> RS[10]: bucketCols=[[VALUE._col0]],numBuckets=-1
> TS[1]: bucketCols=[[]],numBuckets=-1
> RS[8]: bucketCols=[[VALUE._col0]],numBuckets=-1
> TS[0]: bucketCols=[[]],numBuckets=-1
> RS[4]: bucketCols=[[VALUE._col3]],numBuckets=-1
> {code}
> This is incorrect:
> Join[9] joins ca join (cs join cust). In this case both sides of join have a 
> '_col0' column. The reverse mapping of trait propagation relies on 
> ExprNodeColumnDesc.isSame; since this doesn't account for the tabAlias we end 
> up with Join[9] being bucketed on cs_sold_date_sk; Join[12] has the same 
> issue, only compounds the error.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6955) ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait Propagation in Joins

2014-04-22 Thread Harish Butani (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6955:


Status: Patch Available  (was: Open)

> ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait 
> Propagation in Joins
> 
>
> Key: HIVE-6955
> URL: https://issues.apache.org/jira/browse/HIVE-6955
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-6955.1.patch
>
>
> For tpcds Q15:
> {code}
> explain
> select ca_zip, sum(cs_sales_price)
> from catalog_sales, customer, customer_address, date_dim
> where catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
>   and customer.c_current_addr_sk = customer_address.ca_address_sk
>   and (substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
>   '85392', '85460', '80348', '81792')
>or ca_state in ('CA','WA','GA')
>or cs_sales_price > 500)
>   and catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
>   and d_qoy = 2 and d_year = 2001
> group by ca_zip
> order by ca_zip
> limit 100;
> {code}
> The Traits setup for the Operators are:
> {code}
> FIL[23]: bucketCols=[[]],numBuckets=-1
> RS[11]: bucketCols=[[VALUE._col0]],numBuckets=-1
> JOIN[12]: bucketCols=[[_col71], [_col71]],numBuckets=-1
> FIL[13]: bucketCols=[[_col71], [_col71]],numBuckets=-1
> SEL[14]: bucketCols=[[_col71], [_col71]],numBuckets=-1
> GBY[15]: bucketCols=[[_col0]],numBuckets=-1
> RS[16]: bucketCols=[[KEY._col0]],numBuckets=-1
> GBY[17]: bucketCols=[[_col0]],numBuckets=-1
> SEL[18]: bucketCols=[[_col0]],numBuckets=-1
> LIM[21]: bucketCols=[[_col0]],numBuckets=-1
> FS[22]: bucketCols=[[_col0]],numBuckets=-1
> TS[3]: bucketCols=[[]],numBuckets=-1
> RS[5]: bucketCols=[[VALUE._col0]],numBuckets=-1
> JOIN[6]: bucketCols=[[_col3], [_col36]],numBuckets=-1
> RS[7]: bucketCols=[[VALUE._col40]],numBuckets=-1
> JOIN[9]: bucketCols=[[_col40], [_col0]],numBuckets=-1
> RS[10]: bucketCols=[[VALUE._col0]],numBuckets=-1
> TS[1]: bucketCols=[[]],numBuckets=-1
> RS[8]: bucketCols=[[VALUE._col0]],numBuckets=-1
> TS[0]: bucketCols=[[]],numBuckets=-1
> RS[4]: bucketCols=[[VALUE._col3]],numBuckets=-1
> {code}
> This is incorrect:
> Join[9] joins ca join (cs join cust). In this case both sides of join have a 
> '_col0' column. The reverse mapping of trait propagation relies on 
> ExprNodeColumnDesc.isSame; since this doesn't account for the tabAlias we end 
> up with Join[9] being bucketed on cs_sold_date_sk; Join[12] has the same 
> issue, only compounds the error.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle

2014-04-22 Thread Sergey Shelukhin (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6945:
---

Attachment: HIVE-6945.patch

> issues with dropping partitions on Oracle
> -
>
> Key: HIVE-6945
> URL: https://issues.apache.org/jira/browse/HIVE-6945
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6945.patch
>
>
> 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
> translated by DN into decimal rather than long. This appears to be specific 
> to some cases because it seemed to have worked before (different version of 
> Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
> 2) When partition dropping code falls back to JDO, it creates objects to 
> return, then drops partitions. It appears that dropping makes DN objects 
> invalid. We create metastore partition objects out of DN objects before drop, 
> however the list of partition column values is re-used, rather than copied, 
> into these. DN appears to clear this list during drop, so the returned object 
> becomes invalid and the exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle

2014-04-22 Thread Sergey Shelukhin (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6945:
---

Status: Patch Available  (was: Open)

> issues with dropping partitions on Oracle
> -
>
> Key: HIVE-6945
> URL: https://issues.apache.org/jira/browse/HIVE-6945
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6945.patch
>
>
> 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
> translated by DN into decimal rather than long. This appears to be specific 
> to some cases because it seemed to have worked before (different version of 
> Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
> 2) When partition dropping code falls back to JDO, it creates objects to 
> return, then drops partitions. It appears that dropping makes DN objects 
> invalid. We create metastore partition objects out of DN objects before drop, 
> however the list of partition column values is re-used, rather than copied, 
> into these. DN appears to clear this list during drop, so the returned object 
> becomes invalid and the exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6955) ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait Propagation in Joins

2014-04-22 Thread Harish Butani (JIRA)

Harish Butani created HIVE-6955:
---

 Summary: ExprNodeColDesc isSame doesn't account for tabAlias: this 
affects trait Propagation in Joins
 Key: HIVE-6955
 URL: https://issues.apache.org/jira/browse/HIVE-6955
 Project: Hive
  Issue Type: Bug
Reporter: Harish Butani


For tpcds Q15:
{code}
explain
select ca_zip, sum(cs_sales_price)
from catalog_sales, customer, customer_address, date_dim
where catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
  and customer.c_current_addr_sk = customer_address.ca_address_sk
  and (substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
  '85392', '85460', '80348', '81792')
   or ca_state in ('CA','WA','GA')
   or cs_sales_price > 500)
  and catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
  and d_qoy = 2 and d_year = 2001
group by ca_zip
order by ca_zip
limit 100;
{code}

The Traits setup for the Operators are:
{code}
FIL[23]: bucketCols=[[]],numBuckets=-1
RS[11]: bucketCols=[[VALUE._col0]],numBuckets=-1
JOIN[12]: bucketCols=[[_col71], [_col71]],numBuckets=-1
FIL[13]: bucketCols=[[_col71], [_col71]],numBuckets=-1
SEL[14]: bucketCols=[[_col71], [_col71]],numBuckets=-1
GBY[15]: bucketCols=[[_col0]],numBuckets=-1
RS[16]: bucketCols=[[KEY._col0]],numBuckets=-1
GBY[17]: bucketCols=[[_col0]],numBuckets=-1
SEL[18]: bucketCols=[[_col0]],numBuckets=-1
LIM[21]: bucketCols=[[_col0]],numBuckets=-1
FS[22]: bucketCols=[[_col0]],numBuckets=-1
TS[3]: bucketCols=[[]],numBuckets=-1
RS[5]: bucketCols=[[VALUE._col0]],numBuckets=-1
JOIN[6]: bucketCols=[[_col3], [_col36]],numBuckets=-1
RS[7]: bucketCols=[[VALUE._col40]],numBuckets=-1
JOIN[9]: bucketCols=[[_col40], [_col0]],numBuckets=-1
RS[10]: bucketCols=[[VALUE._col0]],numBuckets=-1
TS[1]: bucketCols=[[]],numBuckets=-1
RS[8]: bucketCols=[[VALUE._col0]],numBuckets=-1
TS[0]: bucketCols=[[]],numBuckets=-1
RS[4]: bucketCols=[[VALUE._col3]],numBuckets=-1
{code}

This is incorrect:
Join[9] joins ca join (cs join cust). In this case both sides of join have a 
'_col0' column. The reverse mapping of trait propagation relies on 
ExprNodeColumnDesc.isSame; since this doesn't account for the tabAlias we end 
up with Join[9] being bucketed on cs_sold_date_sk; Join[12] has the same issue, 
only compounds the error.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-6955) ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait Propagation in Joins

2014-04-22 Thread Harish Butani (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani reassigned HIVE-6955:
---

Assignee: Harish Butani

> ExprNodeColDesc isSame doesn't account for tabAlias: this affects trait 
> Propagation in Joins
> 
>
> Key: HIVE-6955
> URL: https://issues.apache.org/jira/browse/HIVE-6955
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
>
> For tpcds Q15:
> {code}
> explain
> select ca_zip, sum(cs_sales_price)
> from catalog_sales, customer, customer_address, date_dim
> where catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
>   and customer.c_current_addr_sk = customer_address.ca_address_sk
>   and (substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
>   '85392', '85460', '80348', '81792')
>or ca_state in ('CA','WA','GA')
>or cs_sales_price > 500)
>   and catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
>   and d_qoy = 2 and d_year = 2001
> group by ca_zip
> order by ca_zip
> limit 100;
> {code}
> The Traits setup for the Operators are:
> {code}
> FIL[23]: bucketCols=[[]],numBuckets=-1
> RS[11]: bucketCols=[[VALUE._col0]],numBuckets=-1
> JOIN[12]: bucketCols=[[_col71], [_col71]],numBuckets=-1
> FIL[13]: bucketCols=[[_col71], [_col71]],numBuckets=-1
> SEL[14]: bucketCols=[[_col71], [_col71]],numBuckets=-1
> GBY[15]: bucketCols=[[_col0]],numBuckets=-1
> RS[16]: bucketCols=[[KEY._col0]],numBuckets=-1
> GBY[17]: bucketCols=[[_col0]],numBuckets=-1
> SEL[18]: bucketCols=[[_col0]],numBuckets=-1
> LIM[21]: bucketCols=[[_col0]],numBuckets=-1
> FS[22]: bucketCols=[[_col0]],numBuckets=-1
> TS[3]: bucketCols=[[]],numBuckets=-1
> RS[5]: bucketCols=[[VALUE._col0]],numBuckets=-1
> JOIN[6]: bucketCols=[[_col3], [_col36]],numBuckets=-1
> RS[7]: bucketCols=[[VALUE._col40]],numBuckets=-1
> JOIN[9]: bucketCols=[[_col40], [_col0]],numBuckets=-1
> RS[10]: bucketCols=[[VALUE._col0]],numBuckets=-1
> TS[1]: bucketCols=[[]],numBuckets=-1
> RS[8]: bucketCols=[[VALUE._col0]],numBuckets=-1
> TS[0]: bucketCols=[[]],numBuckets=-1
> RS[4]: bucketCols=[[VALUE._col3]],numBuckets=-1
> {code}
> This is incorrect:
> Join[9] joins ca join (cs join cust). In this case both sides of join have a 
> '_col0' column. The reverse mapping of trait propagation relies on 
> ExprNodeColumnDesc.isSame; since this doesn't account for the tabAlias we end 
> up with Join[9] being bucketed on cs_sold_date_sk; Join[12] has the same 
> issue, only compounds the error.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6756) alter table set fileformat should set serde too

2014-04-22 Thread Chinna Rao Lalam (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977139#comment-13977139
 ] 

Chinna Rao Lalam commented on HIVE-6756:


With out the patch, current code is taken care for the RC,ORC and PARQUET file 
formats (ALTER TATBLE SET FILEFORMT configuring the proper serde for RC,ORC and 
PARQUET file formats)

TEXTFILE, SEQUENCE file formats are not handled. This patch will address by 
configuring LazySimpleSerde for these file formats.

Apart from this in ALTER TATBLE SET FILEFORMT can use INPUTFORMAT,OUTPUTFORMAT 
classes. In this scenario not sure which serde need to be configure?

If throws exception he cannot use INPUTFORMAT,OUTPUTFORMAT classes in ALTER 
TATBLE SET FILEFORMT.

Any suggestions..

> alter table set fileformat should set serde too
> ---
>
> Key: HIVE-6756
> URL: https://issues.apache.org/jira/browse/HIVE-6756
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-6756.patch
>
>
> Currently doing alter table set fileformat doesn't change the serde. This is 
> unexpected by customers because the serdes are largely file format specific.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (HIVE-6951) Hive 0.13 HiveOutputFormat breaks backwards compatibility

2014-04-22 Thread Ashutosh Chauhan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan reopened HIVE-6951:



> Hive 0.13 HiveOutputFormat breaks backwards compatibility
> -
>
> Key: HIVE-6951
> URL: https://issues.apache.org/jira/browse/HIVE-6951
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Costin Leau
>Priority: Blocker
>
> Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) 
> breaking backwards compatibility with previous releases; the return type of 
> getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter.
> FSRecordWriter introduces one new method on top of RecordWriter however it 
> does not extend the previous interface and it lives in a completely new 
> package.
> Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, 
> code running on HIve 0.13, will break on anything lower than this.
> This could have easily been avoided by extending the existing interface or 
> introducing a new one that RecordWriter could have extended going forward. By 
> changing the signature, the existing contract (and compatibility) has been 
> voided.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HIVE-6951) Hive 0.13 HiveOutputFormat breaks backwards compatibility

2014-04-22 Thread Ashutosh Chauhan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan resolved HIVE-6951.


Resolution: Fixed

Dupe of HIVE-6952

> Hive 0.13 HiveOutputFormat breaks backwards compatibility
> -
>
> Key: HIVE-6951
> URL: https://issues.apache.org/jira/browse/HIVE-6951
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Costin Leau
>Priority: Blocker
>
> Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) 
> breaking backwards compatibility with previous releases; the return type of 
> getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter.
> FSRecordWriter introduces one new method on top of RecordWriter however it 
> does not extend the previous interface and it lives in a completely new 
> package.
> Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, 
> code running on HIve 0.13, will break on anything lower than this.
> This could have easily been avoided by extending the existing interface or 
> introducing a new one that RecordWriter could have extended going forward. By 
> changing the signature, the existing contract (and compatibility) has been 
> voided.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HIVE-6951) Hive 0.13 HiveOutputFormat breaks backwards compatibility

2014-04-22 Thread Ashutosh Chauhan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan resolved HIVE-6951.


Resolution: Duplicate

> Hive 0.13 HiveOutputFormat breaks backwards compatibility
> -
>
> Key: HIVE-6951
> URL: https://issues.apache.org/jira/browse/HIVE-6951
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Costin Leau
>Priority: Blocker
>
> Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) 
> breaking backwards compatibility with previous releases; the return type of 
> getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter.
> FSRecordWriter introduces one new method on top of RecordWriter however it 
> does not extend the previous interface and it lives in a completely new 
> package.
> Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, 
> code running on HIve 0.13, will break on anything lower than this.
> This could have easily been avoided by extending the existing interface or 
> introducing a new one that RecordWriter could have extended going forward. By 
> changing the signature, the existing contract (and compatibility) has been 
> voided.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6939) TestExecDriver.testMapRedPlan3 fails on hadoop-2

2014-04-22 Thread Ashutosh Chauhan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6939:
---

   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Committed to trunk.

> TestExecDriver.testMapRedPlan3 fails on hadoop-2
> 
>
> Key: HIVE-6939
> URL: https://issues.apache.org/jira/browse/HIVE-6939
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.14.0
>
> Attachments: HIVE-6939.1.patch
>
>
> Passes on hadoop-1, but fails on hadoop-2.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive

2014-04-22 Thread Ashutosh Chauhan (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977120#comment-13977120
 ] 

Ashutosh Chauhan commented on HIVE-5771:


Cool. As you can see from last test run on .8 patch, only handful of failures 
are remaining which needs to be looked into it.

> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, 
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, 
> HIVE-5771.8.patch, HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions 
> are evaluated at runtime. 
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase, 
> however, it is still a runtime evaluation and it doesn't propagate constants 
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6945) issues with dropping partitions on Oracle

2014-04-22 Thread Sergey Shelukhin (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977109#comment-13977109
 ] 

Sergey Shelukhin commented on HIVE-6945:


Updated

> issues with dropping partitions on Oracle
> -
>
> Key: HIVE-6945
> URL: https://issues.apache.org/jira/browse/HIVE-6945
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
> translated by DN into decimal rather than long. This appears to be specific 
> to some cases because it seemed to have worked before (different version of 
> Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
> 2) When partition dropping code falls back to JDO, it creates objects to 
> return, then drops partitions. It appears that dropping makes DN objects 
> invalid. We create metastore partition objects out of DN objects before drop, 
> however the list of partition column values is re-used, rather than copied, 
> into these. DN appears to clear this list during drop, so the returned object 
> becomes invalid and the exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle

2014-04-22 Thread Sergey Shelukhin (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6945:
---

Description: 
1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
translated by DN into decimal rather than long. This appears to be specific to 
some cases because it seemed to have worked before (different version of 
Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
2) When partition dropping code falls back to JDO, it creates objects to 
return, then drops partitions. It appears that dropping makes DN objects 
invalid. We create metastore partition objects out of DN objects before drop, 
however the list of partition column values is re-used, rather than copied, 
into these. DN appears to clear this list during drop, so the returned object 
becomes invalid and the exception is thrown.

> issues with dropping partitions on Oracle
> -
>
> Key: HIVE-6945
> URL: https://issues.apache.org/jira/browse/HIVE-6945
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
> translated by DN into decimal rather than long. This appears to be specific 
> to some cases because it seemed to have worked before (different version of 
> Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
> 2) When partition dropping code falls back to JDO, it creates objects to 
> return, then drops partitions. It appears that dropping makes DN objects 
> invalid. We create metastore partition objects out of DN objects before drop, 
> however the list of partition column values is re-used, rather than copied, 
> into these. DN appears to clear this list during drop, so the returned object 
> becomes invalid and the exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist

2014-04-22 Thread Alan Gates (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977087#comment-13977087
 ] 

Alan Gates commented on HIVE-6953:
--

[~amareshwari], I can't reproduce this.  Tests pass for me in 0.13 and trunk.  
These tests should be creating the tables they need as part of the tests (see 
TxnHandler.checkQFileTestHack()).  Is there anything in your logs indicating it 
tried to create the tables and failed?  Are you doing anything in your build to 
turn off the hive.in.test config value?  This would prevent the tables being 
created.

> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
>
> When I'm running all tests through the command 'mvn clean install 
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner 
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from 
> transaction database java.sql.SQLSyntaxErrorException: Table/View 
> 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
> Source)
> 
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
>  Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6932) hive README needs update

2014-04-22 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977082#comment-13977082
 ] 

Hive QA commented on HIVE-6932:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12641158/HIVE-6932.1.patch

{color:red}ERROR:{color} -1 due to 43 failed/errored test(s), 5416 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/5/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/5/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 43 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12641158

> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6932) hive README needs update

2014-04-22 Thread Thejas M Nair (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6932:


Attachment: HIVE-6932.2.patch

[~leftylev] Updated patch addressing the comments. Also changed the wording 
regarding query execution.


> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6932) hive README needs update

2014-04-22 Thread Thejas M Nair (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6932:


Attachment: (was: README.txt)

> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6932) hive README needs update

2014-04-22 Thread Thejas M Nair (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6932:


Attachment: (was: HIVE-6932.2.patch)

> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6932) hive README needs update

2014-04-22 Thread Thejas M Nair (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6932:


Attachment: README.txt

> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6932) hive README needs update

2014-04-22 Thread Thejas M Nair (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6932:


Attachment: HIVE-6932.2.patch

> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist

2014-04-22 Thread Alan Gates (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates reassigned HIVE-6953:


Assignee: Alan Gates

> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
>
> When I'm running all tests through the command 'mvn clean install 
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner 
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from 
> transaction database java.sql.SQLSyntaxErrorException: Table/View 
> 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
> Source)
> 
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
>  Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6927) Add support for MSSQL in schematool

2014-04-22 Thread Ashutosh Chauhan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6927:
---

   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Failures are hadoop-2 related. Committed to trunk. Thanks, Deepesh!

> Add support for MSSQL in schematool
> ---
>
> Key: HIVE-6927
> URL: https://issues.apache.org/jira/browse/HIVE-6927
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Deepesh Khandelwal
> Fix For: 0.14.0
>
> Attachments: HIVE-6927.patch
>
>
> Schematool is the preferred way of initializing schema for Hive. Since 
> HIVE-6862 provided the script for MSSQL it would be nice to add the support 
> for it in schematool.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-6893) out of sequence error in HiveMetastore server

2014-04-22 Thread Naveen Gangam (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam reassigned HIVE-6893:
---

Assignee: Naveen Gangam

> out of sequence error in HiveMetastore server
> -
>
> Key: HIVE-6893
> URL: https://issues.apache.org/jira/browse/HIVE-6893
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
>Reporter: Romain Rigaux
>Assignee: Naveen Gangam
>
> Calls listing databases or tables fail. It seems to be a concurrency problem.
> {code}
> 014-03-06 05:34:00,785 ERROR hive.log: 
> org.apache.thrift.TApplicationException: get_databases failed: out of 
> sequence response
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
> at 
> org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
> at 
> org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
> at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
> at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
> at 
> org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
> at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
> at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException

2014-04-22 Thread Lars Francke (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976820#comment-13976820
 ] 

Lars Francke commented on HIVE-2540:


We're still seeing this issue with Hive 0.12 (slightly patched as it's the one 
from CDH5):

{quote}
2014-04-22 16:18:50,304 WARN [main] org.apache.hadoop.mapred.YarnChild: 
Exception running child : java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
processing row 
{"product_id":140719,"active":null,"avg_product_price":null,"base_cats":[[1247,818,821,902,905],[1247,818,821,902,906],[1247,818,821,902,909]]}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error 
while processing row 
{"product_id":140719,"active":null,"avg_product_price":null,"base_cats":[[1247,818,821,902,905],[1247,818,821,902,906],[1247,818,821,902,909]]}
at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
... 8 more
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFExplode.process(GenericUDTFExplode.java:92)
at 
org.apache.hadoop.hive.ql.exec.UDTFOperator.processOp(UDTFOperator.java:113)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at 
org.apache.hadoop.hive.ql.exec.LateralViewForwardOperator.processOp(LateralViewForwardOperator.java:37)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)
... 9 more
{quote}

I haven't had time to check whether this patch is actually included but I'd 
expect it to be. Shall I open a new issue?

> LATERAL VIEW with EXPLODE produces ConcurrentModificationException
> --
>
> Key: HIVE-2540
> URL: https://issues.apache.org/jira/browse/HIVE-2540
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.7.1, 0.9.0
>Reporter: David Phillips
>Assignee: Navis
> Fix For: 0.10.0
>
> Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.1.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.2.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.3.patch
>
>
> The following produces {{ConcurrentModificationException}} on the {{for}} 
> loop inside EXPLODE:
> {code}
> create table foo as select array(1, 2) a from src limit 1;
> select a, x.b from foo lateral view explode(a) x as b;
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Hive Contributor

2014-04-22 Thread Naveen Gangam

Thank you Ashutosh.


On Tue, Apr 22, 2014 at 12:44 AM, Ashutosh Chauhan wrote:

> Welcome aboard, Naveen!
> I have added you as contributor to project. Looking forward to your
> contributions to Hive.
>
> Ashutosh
>
>
> On Mon, Apr 21, 2014 at 7:18 PM, Naveen Gangam wrote:
>
>> Dear Hive PMC,
>> I would like to contribute to the HIVE community. Could you please grant
>> me
>> the contributor role?
>>
>> My apache username is "ngangam". Thank you in advance and I am looking
>> forward to becoming a part of the Hive community.
>>
>> --
>>
>> Thanks,
>> Naveen :)
>>
>
>


-- 

Thanks,
Naveen :)

[jira] [Commented] (HIVE-6927) Add support for MSSQL in schematool

2014-04-22 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976762#comment-13976762
 ] 

Hive QA commented on HIVE-6927:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12640686/HIVE-6927.patch

{color:red}ERROR:{color} -1 due to 58 failed/errored test(s), 5483 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testNameMethods
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartition
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/4/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job

[jira] [Created] (HIVE-6954) After ALTER FILEFORMAT, DESCRIBE throwing exception

2014-04-22 Thread Chinna Rao Lalam (JIRA)

Chinna Rao Lalam created HIVE-6954:
--

 Summary: After ALTER FILEFORMAT, DESCRIBE throwing exception
 Key: HIVE-6954
 URL: https://issues.apache.org/jira/browse/HIVE-6954
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.14.0
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam


After ALTER TABLE FILEFORMAT, DESCRIBE EXTENDS TABLE throwing exception.

{quote}

CREATE TABLE alter_test ( id int, str string,mp MAP,lst 
ARRAY,strct STRUCT) PARTITIONED BY (part string) 
STORED AS TEXTFILE;

DESCRIBE EXTENDED alter_test;

ALTER TABLE alter_test SET FILEFORMAT PARQUET;

DESCRIBE EXTENDED alter_test;

SELECT * FROM alter_test;

{quote}


hive> DESCRIBE EXTENDED alter_test;
FAILED: SemanticException : Table not found alter_test
hive> SELECT * FROM alter_test;
FAILED: SemanticException must specify an InputFormat class



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist

2014-04-22 Thread Amareshwari Sriramadasu (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976740#comment-13976740
 ] 

Amareshwari Sriramadasu commented on HIVE-6953:
---

There are no failures in trunk, all tests pass. [~rhbutani], do you think 
anything is missing in branch-0.13? Looking at commits, i couldnt figure out.


> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Amareshwari Sriramadasu
>
> When I'm running all tests through the command 'mvn clean install 
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner 
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from 
> transaction database java.sql.SQLSyntaxErrorException: Table/View 
> 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
> Source)
> 
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
>  Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist

2014-04-22 Thread Amareshwari Sriramadasu (JIRA)

Amareshwari Sriramadasu created HIVE-6953:
-

 Summary: All CompactorTest failing with Table/View 'NEXT_TXN_ID' 
does not exist
 Key: HIVE-6953
 URL: https://issues.apache.org/jira/browse/HIVE-6953
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.13.0
Reporter: Amareshwari Sriramadasu


When I'm running all tests through the command 'mvn clean install -Phadoop-1', 
all CompactorTest classes TestInitiator, TestWorker, TestCleaner fail with 
following exception :

{noformat}
org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from 
transaction database java.sql.SQLSyntaxErrorException: Table/View 'NEXT_TXN_ID' 
does not exist.
at 
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
Source)
at 
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
Source)
at 
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
Source)


Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
at 
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at 
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
 Source)

{noformat}

This is happening on branch-0.13. Has anyone faced this problem?

[~owen.omalley] or someone else help me solve this. Do i have to set anything?






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6947) More fixes for tests on hadoop-2

2014-04-22 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976633#comment-13976633
 ] 

Hive QA commented on HIVE-6947:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12641157/HIVE-6947.patch

{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 5483 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/3/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/3/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 53 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12641157

> More fixe

Re: 49 config params without descriptions

2014-04-22 Thread Lefty Leverenz

Found two more from HIVE-5522
 (also HIVE-6098 , Merge
Tez branch into trunk) so the current total is 51 configs that don't have
descriptions in 0.13.0:

*Release 13 *

hive.compute.splits.in.am

hive.rpc.query.plan


But these both have comments in HiveConf.java that can be used as
descriptions, although they aren't included in hive-default.xml.template.
 I missed them because I was working from the patch for
HIVE-6037 and
Navis had used the HiveConf comments for descriptions.  (That means there
could be more parameters missing from the 0.13.0 template file.)



-- Lefty


On Mon, Apr 14, 2014 at 1:53 AM, Lefty Leverenz wrote:

> Here's a list of 49 configuration parameters in RC0 (and trunk) that don't
> have descriptions in hive-default.xml.template:
>
>
> *Release 1 or 2 *
>
> hive.exec.submitviachild
>
> hive.metastore.metadb.dir
>
> hive.jar.path
>
> hive.aux.jars.path
>
> hive.table.name
>
> hive.partition.name
>
> hive.alias
>
>
> *Release 3 *
>
> hive.cli.errors.ignore
>
>
> *Release 4 *
>
> hive.added.files.path
>
> hive.added.jars.path
>
>
> *Release 5 *
>
> hive.intermediate.compression.codec
>
> hive.intermediate.compression.type
>
> hive.added.archives.path
>
>
> *Release 6 *
>
> hive.metastore.archive.intermediate.archived
>
> hive.metastore.archive.intermediate.extracted
>
> hive.mapred.partitioner
>
> hive.exec.script.trust
>
> hive.hadoop.supports.splittable.combineinputformat
>
>
> *Release 7 *
>
> hive.lockmgr.zookeeper.default.partition.name
>
> hive.metastore.fs.handler.class
>
> hive.query.result.fileformat
>
> hive.hashtable.initialCapacity
>
> hive.hashtable.loadfactor
>
> hive.debug.localtask
>
> hive.lock.manager
>
> hive.outerjoin.supports.filters
>
> hive.semantic.analyzer.hook
>
>
> *Release 8 *
>
> hive.exec.job.debug.timeout
>
> hive.exec.tasklog.debug.timeout
>
> hive.merge.rcfile.block.level
>
> hive.merge.input.format.block.level
>
> hive.merge.current.job.has.dynamic.partitions
>
> hive.stats.collect.rawdatasize
>
>
> *Release 8.1 *
>
> hive.optimize.metadataonly
>
>
> *Release 9 *
>
>
> *Release 10 *
>
>
> *Release 11 *
>
> hive.exec.rcfile.use.sync.cache
>
> hive.stats.key.prefix--- *internal*
>
>
> *Release 12 *
>
> hive.scratch.dir.permission
>
> datanucleus.fixedDatastore
>
> datanucleus.rdbms.useLegacyNativeValueStrategy
>
> hive.optimize.sampling.orderby --- *internal?*
>
> hive.optimize.sampling.orderby.number
>
> hive.optimize.sampling.orderby.percent
>
> hive.server2.authentication.ldap.Domain
>
> hive.server2.session.hook
>
> hive.typecheck.on.insert
>
>
> *Release 13 *
>
> hive.metastore.expression.proxy
>
> hive.txn.manager
>
> hive.stageid.rearrange
>
> hive.explain.dependency.append.tasktype
>
>
>
> What's the best way to deal with these?
>
>1. Ignore them (or identify those that can be ignored).
>2. Add some descriptions in Hive 0.13.0 RC1.
>3. Deal with them after 
> HIVE-6037gets committed.
>   - Try to cover all of them by Hive 0.14.0:
>   - Put the list in a JIRA and create a common HiveConf.java patch,
>  which can be appended until release 0.14.0 is ready.
>  - Accumulate descriptions in JIRA comments, then create a patch
>  from the comments.
>   - Deal with them as soon as possible:
>   - Put the list in an umbrella JIRA and use sub-task JIRAs to add
>  descriptions individually or in small groups.
>  4. Deal with them in the wiki, then patch HiveConf.java before
>release 0.14.0.
>5. [Your idea goes here.]
>
>
> -- Lefty
>

[jira] [Commented] (HIVE-6952) Hive 0.13 HiveOutputFormat breaks backwards compatibility

2014-04-22 Thread Costin Leau (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976591#comment-13976591
 ] 

Costin Leau commented on HIVE-6952:
---

Actually taking a closer look indicates that FSRecordWriter and RecordWriter 
are identical - they both implement the same two methods, with the _exact_ same 
signature which makes the breaking change even more puzzling.

> Hive 0.13 HiveOutputFormat breaks backwards compatibility
> -
>
> Key: HIVE-6952
> URL: https://issues.apache.org/jira/browse/HIVE-6952
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Costin Leau
>Priority: Blocker
>
> Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) 
> breaking backwards compatibility with previous releases; the return type of 
> getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter.
> FSRecordWriter introduces one new method on top of RecordWriter however it 
> does not extend the previous interface and it lives in a completely new 
> package.
> Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, 
> code running on HIve 0.13, will break on anything lower than this.
> This could have easily been avoided by extending the existing interface or 
> introducing a new one that RecordWriter could have extended going forward. By 
> changing the signature, the existing contract (and compatibility) has been 
> voided.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6952) Hive 0.13 HiveOutputFormat breaks backwards compatibility

2014-04-22 Thread Costin Leau (JIRA)

Costin Leau created HIVE-6952:
-

 Summary: Hive 0.13 HiveOutputFormat breaks backwards compatibility
 Key: HIVE-6952
 URL: https://issues.apache.org/jira/browse/HIVE-6952
 Project: Hive
  Issue Type: Bug
  Components: File Formats, Serializers/Deserializers
Affects Versions: 0.13.0
Reporter: Costin Leau
Priority: Blocker


Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) 
breaking backwards compatibility with previous releases; the return type of 
getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter.

FSRecordWriter introduces one new method on top of RecordWriter however it does 
not extend the previous interface and it lives in a completely new package.
Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, 
code running on HIve 0.13, will break on anything lower than this.

This could have easily been avoided by extending the existing interface or 
introducing a new one that RecordWriter could have extended going forward. By 
changing the signature, the existing contract (and compatibility) has been 
voided.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6951) Hive 0.13 HiveOutputFormat breaks backwards compatibility

2014-04-22 Thread Costin Leau (JIRA)

Costin Leau created HIVE-6951:
-

 Summary: Hive 0.13 HiveOutputFormat breaks backwards compatibility
 Key: HIVE-6951
 URL: https://issues.apache.org/jira/browse/HIVE-6951
 Project: Hive
  Issue Type: Bug
  Components: File Formats, Serializers/Deserializers
Affects Versions: 0.13.0
Reporter: Costin Leau
Priority: Blocker


Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) 
breaking backwards compatibility with previous releases; the return type of 
getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter.

FSRecordWriter introduces one new method on top of RecordWriter however it does 
not extend the previous interface and it lives in a completely new package.
Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, 
code running on HIve 0.13, will break on anything lower than this.

This could have easily been avoided by extending the existing interface or 
introducing a new one that RecordWriter could have extended going forward. By 
changing the signature, the existing contract (and compatibility) has been 
voided.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5538) Turn on vectorization by default.

2014-04-22 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976533#comment-13976533
 ] 

Hive QA commented on HIVE-5538:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12641155/HIVE-5538.3.patch

{color:red}ERROR:{color} -1 due to 71 failed/errored test(s), 5417 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_diff_part_cols
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_diff_part_cols2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_create
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partInit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quote1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_

[jira] [Commented] (HIVE-2621) Allow multiple group bys with the same input data and spray keys to be run on the same reducer.

2014-04-22 Thread Lefty Leverenz (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976525#comment-13976525
 ] 

Lefty Leverenz commented on HIVE-2621:
--

Asking again:  Is the definition of *hive.multigroupby.singlereducer* correct 
or was it just held over from *hive.multigroupby.singlemr*?  (See previous 
comment.)

> Allow multiple group bys with the same input data and spray keys to be run on 
> the same reducer.
> ---
>
> Key: HIVE-2621
> URL: https://issues.apache.org/jira/browse/HIVE-2621
> Project: Hive
>  Issue Type: New Feature
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Fix For: 0.9.0
>
> Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2621.D567.1.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-2621.D567.2.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-2621.D567.3.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-2621.D567.4.patch, HIVE-2621.1.patch.txt
>
>
> Currently, when a user runs a query, such as a multi-insert, where each 
> insertion subclause consists of a simple query followed by a group by, the 
> group bys for each clause are run on a separate reducer.  This requires 
> writing the data for each group by clause to an intermediate file, and then 
> reading it back.  This uses a significant amount of the total CPU consumed by 
> the query for an otherwise simple query.
> If the subclauses are grouped by their distinct expressions and group by 
> keys, with all of the group by expressions for a group of subclauses run on a 
> single reducer, this would reduce the amount of reading/writing to 
> intermediate files for some queries.
> To do this, for each group of subclauses, in the mapper we would execute a 
> the filters for each subclause 'or'd together (provided each subclause has a 
> filter) followed by a reduce sink.  In the reducer, the child operators would 
> be each subclauses filter followed by the group by and any subsequent 
> operations.
> Note that this would require turning off map aggregation, so we would need to 
> make using this type of plan configurable.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6932) hive README needs update

2014-04-22 Thread Lefty Leverenz (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976507#comment-13976507
 ] 

Lefty Leverenz commented on HIVE-6932:
--

Nano-nit #2:  need another space before "frameworks."

{noformat}
+* Query execution via Apache Hadoop MapReduce and using Apache Tez
+ frameworks.
{noformat}

Also a question:  Is "QL" the proper name for the language or "HiveQL"?  The 
README says HiveQL twice in the Getting Started section.  The wiki is 
inconsistent with QL, HiveQL, and Hive QL.

> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6932) hive README needs update

2014-04-22 Thread Lefty Leverenz (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976493#comment-13976493
 ] 

Lefty Leverenz commented on HIVE-6932:
--

Nano-nit:  an extra space before "your own upgrade script."

{noformat}
+- We have provided upgrade scripts for MySQL, PostgreSQL, Oracle,
+  Microsoft SQL Server, and Derby databases. If you are using a
+  different database for your MetaStore you will need to provide
+   your own upgrade script.
{noformat}

> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

< 1 2

101 - 139 of 139 matches

Mail list logo