[jira] [Commented] (HIVE-6473) Allow writing HFiles via HBaseStorageHandler table

[Swarnim Kulkarni (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017420#comment-14017420
 ] 

Swarnim Kulkarni commented on HIVE-6473:


[~ndimiduk] I ran the test locally with the latest patch and was able to 
reproduce the failure.

{noformat}
Failed tests: 
  TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk:92->runTest:122 Unexpected 
exception

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 3:01.311s
[INFO] Finished at: Wed Jun 04 01:26:30 CDT 2014
[INFO] Final Memory: 27M/87M
[INFO] 
{noformat}

Also as a side note, it seems like the latest patch does not apply cleanly. May 
be it would be regnerated with "--no-prefix" option.

> Allow writing HFiles via HBaseStorageHandler table
> --
>
> Key: HIVE-6473
> URL: https://issues.apache.org/jira/browse/HIVE-6473
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: HIVE-6473.0.patch.txt, HIVE-6473.1.patch, 
> HIVE-6473.1.patch.txt, HIVE-6473.2.patch, HIVE-6473.3.patch, 
> HIVE-6473.4.patch, HIVE-6473.5.patch
>
>
> Generating HFiles for bulkload into HBase could be more convenient. Right now 
> we require the user to register a new table with the appropriate output 
> format. This patch allows the exact same functionality, but through an 
> existing table managed by the HBaseStorageHandler.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6473) Allow writing HFiles via HBaseStorageHandler table

[Swarnim Kulkarni (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017421#comment-14017421
 ] 

Swarnim Kulkarni commented on HIVE-6473:


You might need to rebase the patch with master to reproduce the failure locally.

> Allow writing HFiles via HBaseStorageHandler table
> --
>
> Key: HIVE-6473
> URL: https://issues.apache.org/jira/browse/HIVE-6473
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: HIVE-6473.0.patch.txt, HIVE-6473.1.patch, 
> HIVE-6473.1.patch.txt, HIVE-6473.2.patch, HIVE-6473.3.patch, 
> HIVE-6473.4.patch, HIVE-6473.5.patch
>
>
> Generating HFiles for bulkload into HBase could be more convenient. Right now 
> we require the user to register a new table with the appropriate output 
> format. This patch allows the exact same functionality, but through an 
> existing table managed by the HBaseStorageHandler.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-7166) Vectorization with UDFs returns incorrect results

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan reassigned HIVE-7166:
---

Assignee: Hari Sankar Sivarama Subramaniyan

> Vectorization with UDFs returns incorrect results
> -
>
> Key: HIVE-7166
> URL: https://issues.apache.org/jira/browse/HIVE-7166
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, UDF, Vectorization
>Affects Versions: 0.13.0
> Environment: Hive 0.13 with Hadoop 2.4 on a 3 node cluster 
>Reporter: Benjamin Bowman
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Minor
>
> Using BETWEEN, a custom UDF, and vectorized query execution yields incorrect 
> query results. 
> Example Query:  SELECT column_1 FROM table_1 WHERE column_1 BETWEEN (UDF_1 - 
> X) and UDF_1
> The following test scenario will reproduce the problem:
> TEST UDF (SIMPLE FUNCTION THAT TAKES NO ARGUMENTS AND RETURNS 1):  
> package com.test;
> import org.apache.hadoop.hive.ql.exec.Description;
> import org.apache.hadoop.hive.ql.exec.UDF;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import java.lang.String;
> import java.lang.*;
> public class tenThousand extends UDF {
>   private final LongWritable result = new LongWritable();
>   public LongWritable evaluate() {
> result.set(1);
> return result;
>   }
> }
> TEST DATA (test.input):
> 1|CBCABC|12
> 2|DBCABC|13
> 3|EBCABC|14
> 4|ABCABC|15
> 5|BBCABC|16
> 6|CBCABC|17
> CREATING ORC TABLE:
> 0: jdbc:hive2://server:10002/db> create table testTabOrc (first bigint, 
> second varchar(20), third int) partitioned by (range int) clustered by 
> (first) sorted by (first) into 8 buckets stored as orc tblproperties 
> ("orc.compress" = "SNAPPY", "orc.index" = "true");
> CREATE LOADING TABLE:
> 0: jdbc:hive2://server:10002/db> create table loadingDir (first bigint, 
> second varchar(20), third int) partitioned by (range int) row format 
> delimited fields terminated by '|' stored as textfile;
> COPY IN DATA:
> [root@server]#  hadoop fs -copyFromLocal /tmp/test.input /db/loading/.
> ORC DATA:
> [root@server]#  beeline -u jdbc:hive2://server:10002/db -n root --hiveconf 
> hive.exec.dynamic.partition.mode=nonstrict --hiveconf 
> hive.enforce.sorting=true -e "insert into table testTabOrc partition(range) 
> select * from loadingDir;"
> LOAD TEST FUNCTION:
> 0: jdbc:hive2://server:10002/db>  add jar /opt/hadoop/lib/testFunction.jar
> 0: jdbc:hive2://server:10002/db>  create temporary function ten_thousand as 
> 'com.test.tenThousand';
> TURN OFF VECTORIZATION:
> 0: jdbc:hive2://server:10002/db>  set hive.vectorized.execution.enabled=false;
> QUERY (RESULTS AS EXPECTED):
> 0: jdbc:hive2://server:10002/db> select first from testTabOrc where first 
> between ten_thousand()-1 and ten_thousand()-9995;
> ++
> | first  |
> ++
> | 1  |
> | 2  |
> | 3  |
> ++
> 3 rows selected (15.286 seconds)
> TURN ON VECTORIZATION:
> 0: jdbc:hive2://server:10002/db>  set hive.vectorized.execution.enabled=true;
> QUERY AGAIN (WRONG RESULTS):
> 0: jdbc:hive2://server:10002/db> select first from testTabOrc where first 
> between ten_thousand()-1 and ten_thousand()-9995;
> ++
> | first  |
> ++
> ++
> No rows selected (17.763 seconds)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7170) Fix display_colstats_tbllvl.q in trunk

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-7170:
---

   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Navis!

> Fix display_colstats_tbllvl.q in trunk
> --
>
> Key: HIVE-7170
> URL: https://issues.apache.org/jira/browse/HIVE-7170
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Prasanth J
>Assignee: Navis
> Fix For: 0.14.0
>
> Attachments: HIVE-7170.1.patch.txt
>
>
> display_colstats_tbllvl.q test case fails will NPE. Following is the exception
> {code}
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.metastore.StatObjectConverter.fillColumnStatisticsData(StatObjectConverter.java:369)
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.makeColumnStats(MetaStoreDirectSql.java:973)
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:888)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:5769)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:5766)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2208)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:5766)
> at 
> org.apache.hadoop.hive.metastore.VerifyingObjectStore.getTableColumnStatistics(VerifyingObjectStore.java:103)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.getTableColumnStatistics(Unknown Source)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:3556)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
> at com.sun.proxy.$Proxy11.get_table_statistics_req(Unknown Source)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:1224)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6473) Allow writing HFiles via HBaseStorageHandler table

[Swarnim Kulkarni (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017386#comment-14017386
 ] 

Swarnim Kulkarni commented on HIVE-6473:


[~ndimiduk] I'll try running the test locally as well and then post back.

> Allow writing HFiles via HBaseStorageHandler table
> --
>
> Key: HIVE-6473
> URL: https://issues.apache.org/jira/browse/HIVE-6473
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: HIVE-6473.0.patch.txt, HIVE-6473.1.patch, 
> HIVE-6473.1.patch.txt, HIVE-6473.2.patch, HIVE-6473.3.patch, 
> HIVE-6473.4.patch, HIVE-6473.5.patch
>
>
> Generating HFiles for bulkload into HBase could be more convenient. Right now 
> we require the user to register a new table with the appropriate output 
> format. This patch allows the exact same functionality, but through an 
> existing table managed by the HBaseStorageHandler.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7117) Partitions not inheriting table permissions after alter rename partition

[Swarnim Kulkarni (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017382#comment-14017382
 ] 

Swarnim Kulkarni commented on HIVE-7117:


+1 from me as well (non-binding)

> Partitions not inheriting table permissions after alter rename partition
> 
>
> Key: HIVE-7117
> URL: https://issues.apache.org/jira/browse/HIVE-7117
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: HIVE-7117.2.patch, HIVE-7117.3.patch, HIVE-7117.4.patch, 
> HIVE-7117.5.patch, HIVE-7117.6.patch, HIVE-7117.7.patch, HIVE-7117.8.patch, 
> HIVE-7117.patch
>
>
> On altering/renaming a partition it must inherit permission of the parent 
> directory, if the flag hive.warehouse.subdir.inherit.perms is set.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7117) Partitions not inheriting table permissions after alter rename partition

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017380#comment-14017380
 ] 

Szehon Ho commented on HIVE-7117:
-

Thanks.  +1 (non-binding)

> Partitions not inheriting table permissions after alter rename partition
> 
>
> Key: HIVE-7117
> URL: https://issues.apache.org/jira/browse/HIVE-7117
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: HIVE-7117.2.patch, HIVE-7117.3.patch, HIVE-7117.4.patch, 
> HIVE-7117.5.patch, HIVE-7117.6.patch, HIVE-7117.7.patch, HIVE-7117.8.patch, 
> HIVE-7117.patch
>
>
> On altering/renaming a partition it must inherit permission of the parent 
> directory, if the flag hive.warehouse.subdir.inherit.perms is set.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7143) Add Streaming support in Windowing mode for more UDAFs (min/max, lead/lag, fval/lval)

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017374#comment-14017374
 ] 

Hive QA commented on HIVE-7143:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12647955/HIVE-7143.3.patch

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 5530 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_display_colstats_tbllvl
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/382/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/382/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-382/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12647955

> Add Streaming support in Windowing mode for more UDAFs (min/max, lead/lag, 
> fval/lval)
> -
>
> Key: HIVE-7143
> URL: https://issues.apache.org/jira/browse/HIVE-7143
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-7143.1.patch, HIVE-7143.3.patch
>
>
> Provided implementations for Streaming for the above fns.
> Min/Max based on Alg by Daniel Lemire: 
> http://www.archipel.uqam.ca/309/1/webmaximinalgo.pdf



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-4867) Deduplicate columns appearing in both the key list and value list of ReduceSinkOperator

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-4867:


Attachment: HIVE-4867.4.patch.txt

> Deduplicate columns appearing in both the key list and value list of 
> ReduceSinkOperator
> ---
>
> Key: HIVE-4867
> URL: https://issues.apache.org/jira/browse/HIVE-4867
> Project: Hive
>  Issue Type: Improvement
>Reporter: Yin Huai
>Assignee: Navis
> Attachments: HIVE-4867.1.patch.txt, HIVE-4867.2.patch.txt, 
> HIVE-4867.3.patch.txt, HIVE-4867.4.patch.txt, source_only.txt
>
>
> A ReduceSinkOperator emits data in the format of keys and values. Right now, 
> a column may appear in both the key list and value list, which result in 
> unnecessary overhead for shuffling. 
> Example:
> We have a query shown below ...
> {code:sql}
> explain select ss_ticket_number from store_sales cluster by ss_ticket_number;
> {\code}
> The plan is ...
> {code}
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-0 is a root stage
> STAGE PLANS:
>   Stage: Stage-1
> Map Reduce
>   Alias -> Map Operator Tree:
> store_sales 
>   TableScan
> alias: store_sales
> Select Operator
>   expressions:
> expr: ss_ticket_number
> type: int
>   outputColumnNames: _col0
>   Reduce Output Operator
> key expressions:
>   expr: _col0
>   type: int
> sort order: +
> Map-reduce partition columns:
>   expr: _col0
>   type: int
> tag: -1
> value expressions:
>   expr: _col0
>   type: int
>   Reduce Operator Tree:
> Extract
>   File Output Operator
> compressed: false
> GlobalTableId: 0
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
>   Stage: Stage-0
> Fetch Operator
>   limit: -1
> {\code}
> The column 'ss_ticket_number' is in both the key list and value list of the 
> ReduceSinkOperator. The type of ss_ticket_number is int. For this case, 
> BinarySortableSerDe will introduce 1 byte more for every int in the key. 
> LazyBinarySerDe will also introduce overhead when recording the length of a 
> int. For every int, 10 bytes should be a rough estimation of the size of data 
> emitted from the Map phase. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7135) Fix test fail of TestTezTask.testSubmit

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017305#comment-14017305
 ] 

Hive QA commented on HIVE-7135:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12648079/HIVE-7135.2.patch.txt

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 5510 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_display_colstats_tbllvl
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testDefaults
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/381/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/381/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-381/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12648079

> Fix test fail of TestTezTask.testSubmit
> ---
>
> Key: HIVE-7135
> URL: https://issues.apache.org/jira/browse/HIVE-7135
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Navis
> Fix For: 0.14.0
>
> Attachments: HIVE-7135.1.patch, HIVE-7135.2.patch.txt
>
>
> HIVE-7043 broke a tez test case.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7174) Do not accept string as scale and precision when reading Avro schema

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated HIVE-7174:
-

Fix Version/s: 0.14.0
   Status: Patch Available  (was: Open)

> Do not accept string as scale and precision when reading Avro schema
> 
>
> Key: HIVE-7174
> URL: https://issues.apache.org/jira/browse/HIVE-7174
> Project: Hive
>  Issue Type: Bug
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 0.14.0
>
> Attachments: HIVE-7174.patch
>
>
> I've noticed that the current AvroSerde will happily accept schema that uses 
> string instead of integer for scale and precision, e.g. fragment 
> {{"precision":"4","scale":"1"}} from following table:
> {code}
> CREATE TABLE `avro_dec1`(
>   `name` string COMMENT 'from deserializer',
>   `value` decimal(4,1) COMMENT 'from deserializer')
> COMMENT 'just drop the schema right into the HQL'
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> TBLPROPERTIES (
>   'numFiles'='1',
>   
> 'avro.schema.literal'='{\"namespace\":\"com.howdy\",\"name\":\"some_schema\",\"type\":\"record\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"value\",\"type\":{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":\"4\",\"scale\":\"1\"}}]}'
> );
> {code}
> However the Decimal spec defined in AVRO-1402 requires only integer to be 
> there and hence is allowing only following fragment instead 
> {{"precision":4,"scale":1}} (e.g. no double quotes around numbers).
> As Hive can propagate this incorrect schema to new files and hence creating 
> files with invalid schema, I think that we should alter the behavior and 
> insist on the correct schema.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7174) Do not accept string as scale and precision when reading Avro schema

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated HIVE-7174:
-

Attachment: HIVE-7174.patch

Attaching patch that replaces method call {{getValueAsInt}} that is allowed to 
do type transformations (String -> Integer) with {{getIntValue}} that will fail 
in case that user will use string instead. I've verified that Hive now do not 
accepts the incorrect schema.

> Do not accept string as scale and precision when reading Avro schema
> 
>
> Key: HIVE-7174
> URL: https://issues.apache.org/jira/browse/HIVE-7174
> Project: Hive
>  Issue Type: Bug
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Attachments: HIVE-7174.patch
>
>
> I've noticed that the current AvroSerde will happily accept schema that uses 
> string instead of integer for scale and precision, e.g. fragment 
> {{"precision":"4","scale":"1"}} from following table:
> {code}
> CREATE TABLE `avro_dec1`(
>   `name` string COMMENT 'from deserializer',
>   `value` decimal(4,1) COMMENT 'from deserializer')
> COMMENT 'just drop the schema right into the HQL'
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> TBLPROPERTIES (
>   'numFiles'='1',
>   
> 'avro.schema.literal'='{\"namespace\":\"com.howdy\",\"name\":\"some_schema\",\"type\":\"record\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"value\",\"type\":{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":\"4\",\"scale\":\"1\"}}]}'
> );
> {code}
> However the Decimal spec defined in AVRO-1402 requires only integer to be 
> there and hence is allowing only following fragment instead 
> {{"precision":4,"scale":1}} (e.g. no double quotes around numbers).
> As Hive can propagate this incorrect schema to new files and hence creating 
> files with invalid schema, I think that we should alter the behavior and 
> insist on the correct schema.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7174) Do not accept string as scale and precision when reading Avro schema

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created HIVE-7174:


 Summary: Do not accept string as scale and precision when reading 
Avro schema
 Key: HIVE-7174
 URL: https://issues.apache.org/jira/browse/HIVE-7174
 Project: Hive
  Issue Type: Bug
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho


I've noticed that the current AvroSerde will happily accept schema that uses 
string instead of integer for scale and precision, e.g. fragment 
{{"precision":"4","scale":"1"}} from following table:

{code}
CREATE TABLE `avro_dec1`(
  `name` string COMMENT 'from deserializer',
  `value` decimal(4,1) COMMENT 'from deserializer')
COMMENT 'just drop the schema right into the HQL'
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
  'numFiles'='1',
  
'avro.schema.literal'='{\"namespace\":\"com.howdy\",\"name\":\"some_schema\",\"type\":\"record\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"value\",\"type\":{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":\"4\",\"scale\":\"1\"}}]}'
);
{code}

However the Decimal spec defined in AVRO-1402 requires only integer to be there 
and hence is allowing only following fragment instead 
{{"precision":4,"scale":1}} (e.g. no double quotes around numbers).

As Hive can propagate this incorrect schema to new files and hence creating 
files with invalid schema, I think that we should alter the behavior and insist 
on the correct schema.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6473) Allow writing HFiles via HBaseStorageHandler table

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017258#comment-14017258
 ] 

Nick Dimiduk commented on HIVE-6473:


Ping [~sushanth], [~swarnim] mind taking another look?

> Allow writing HFiles via HBaseStorageHandler table
> --
>
> Key: HIVE-6473
> URL: https://issues.apache.org/jira/browse/HIVE-6473
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: HIVE-6473.0.patch.txt, HIVE-6473.1.patch, 
> HIVE-6473.1.patch.txt, HIVE-6473.2.patch, HIVE-6473.3.patch, 
> HIVE-6473.4.patch, HIVE-6473.5.patch
>
>
> Generating HFiles for bulkload into HBase could be more convenient. Right now 
> we require the user to register a new table with the appropriate output 
> format. This patch allows the exact same functionality, but through an 
> existing table managed by the HBaseStorageHandler.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-2365) SQL support for bulk load into HBase

[Nick Dimiduk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HIVE-2365:
---

Status: Open  (was: Patch Available)

Canceling patch until dependency HIVE-6473 is committed.

> SQL support for bulk load into HBase
> 
>
> Key: HIVE-2365
> URL: https://issues.apache.org/jira/browse/HIVE-2365
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Reporter: John Sichi
>Assignee: Nick Dimiduk
> Fix For: 0.14.0
>
> Attachments: HIVE-2365.2.patch.txt, HIVE-2365.3.patch, 
> HIVE-2365.WIP.00.patch, HIVE-2365.WIP.01.patch, HIVE-2365.WIP.01.patch
>
>
> Support the "as simple as this" SQL for bulk load from Hive into HBase.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6473) Allow writing HFiles via HBaseStorageHandler table

[Nick Dimiduk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HIVE-6473:
---

Attachment: HIVE-6473.5.patch

Test still passes locally for me:

{noformat}
---
Test set: org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver
---
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.33 sec - in 
org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver
{noformat}

Re-attaching patch for build bot.

> Allow writing HFiles via HBaseStorageHandler table
> --
>
> Key: HIVE-6473
> URL: https://issues.apache.org/jira/browse/HIVE-6473
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: HIVE-6473.0.patch.txt, HIVE-6473.1.patch, 
> HIVE-6473.1.patch.txt, HIVE-6473.2.patch, HIVE-6473.3.patch, 
> HIVE-6473.4.patch, HIVE-6473.5.patch
>
>
> Generating HFiles for bulkload into HBase could be more convenient. Right now 
> we require the user to register a new table with the appropriate output 
> format. This patch allows the exact same functionality, but through an 
> existing table managed by the HBaseStorageHandler.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7110) TestHCatPartitionPublish test failure: No FileSystem or scheme: pfile

[David Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017250#comment-14017250
 ] 

David Chen commented on HIVE-7110:
--

[~cwsteinbach] The only other test in hcatalog-core that uses the 
ProxyFileSystem is TestEximSemanticAnalysis, which is currently disabled. 
According to the Git logs, that test was broken (i.e. .java.broken) ever since 
it was checked in when the HCatalog codebase was merged into the Hive codebase.

As for whether TestHCatPartitionPublish ever worked, one indication that it 
worked at one point was the following commit, which apparently fixed HCatalog 
unit tests to run on Windows: 
https://github.com/apache/hive/commit/e884d60a802ee0c76f09c18b08ce93dab7383267

I can use something like git bisect to see whether the test passed at one 
point, but that will be very time consuming. I think that should be done as 
part of a separate task since this test works now with this patch and since I 
will be doing more work involving the HCatalog core tests anyway.

> TestHCatPartitionPublish test failure: No FileSystem or scheme: pfile
> -
>
> Key: HIVE-7110
> URL: https://issues.apache.org/jira/browse/HIVE-7110
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7110.1.patch, HIVE-7110.2.patch, HIVE-7110.3.patch, 
> HIVE-7110.4.patch
>
>
> I got the following TestHCatPartitionPublish test failure when running all 
> unit tests against Hadoop 1. This also appears when testing against Hadoop 2.
> {code}
>  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.06 sec 
> <<< FAILURE! - in org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish
> testPartitionPublish(org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish)
>   Time elapsed: 1.361 sec  <<< ERROR!
> org.apache.hive.hcatalog.common.HCatException: 
> org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output 
> information. Cause : java.io.IOException: No FileSystem for scheme: pfile
> at 
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1443)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
> at 
> org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:212)
> at 
> org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:70)
> at 
> org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.runMRCreateFail(TestHCatPartitionPublish.java:191)
> at 
> org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.testPartitionPublish(TestHCatPartitionPublish.java:155)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7117) Partitions not inheriting table permissions after alter rename partition

[Ashish Kumar Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Kumar Singh updated HIVE-7117:
-

Attachment: HIVE-7117.8.patch

Addressed review comments after rebase. Replied at RB.

> Partitions not inheriting table permissions after alter rename partition
> 
>
> Key: HIVE-7117
> URL: https://issues.apache.org/jira/browse/HIVE-7117
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: HIVE-7117.2.patch, HIVE-7117.3.patch, HIVE-7117.4.patch, 
> HIVE-7117.5.patch, HIVE-7117.6.patch, HIVE-7117.7.patch, HIVE-7117.8.patch, 
> HIVE-7117.patch
>
>
> On altering/renaming a partition it must inherit permission of the parent 
> directory, if the flag hive.warehouse.subdir.inherit.perms is set.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7170) Fix display_colstats_tbllvl.q in trunk

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017170#comment-14017170
 ] 

Hive QA commented on HIVE-7170:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12648076/HIVE-7170.1.patch.txt

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 5585 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadata_only_queries
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_schema_evolution
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/380/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/380/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-380/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12648076

> Fix display_colstats_tbllvl.q in trunk
> --
>
> Key: HIVE-7170
> URL: https://issues.apache.org/jira/browse/HIVE-7170
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Prasanth J
>Assignee: Navis
> Attachments: HIVE-7170.1.patch.txt
>
>
> display_colstats_tbllvl.q test case fails will NPE. Following is the exception
> {code}
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.metastore.StatObjectConverter.fillColumnStatisticsData(StatObjectConverter.java:369)
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.makeColumnStats(MetaStoreDirectSql.java:973)
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:888)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:5769)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:5766)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2208)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:5766)
> at 
> org.apache.hadoop.hive.metastore.VerifyingObjectStore.getTableColumnStatistics(VerifyingObjectStore.java:103)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.getTableColumnStatistics(Unknown Source)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:3556)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
> at com.sun.proxy.$Proxy11.get_table_statistics_req(Unknown Source)
> at 
> org.apache.

Re: Review Request 21846: HIVE-7117: Partitions not inheriting table permissions after alter rename partition

[Ashish Singh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21846/
---

(Updated June 3, 2014, 9:16 p.m.)


Review request for hive and Szehon Ho.


Changes
---

Correcting patch. Ignore previous patch.


Bugs: HIVE-7117
https://issues.apache.org/jira/browse/HIVE-7117


Repository: hive-git


Description
---

Correct FolderPermissionsBase test and change rename method to inherit 
permissions and/or extended ACLs


Inherit permissions from parent while renaming directories or files under hive


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 4591678 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java
 57fe4cf 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
221b010 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c1790b4 

Diff: https://reviews.apache.org/r/21846/diff/


Testing
---

Tested with itests.


Thanks,

Ashish Singh

Re: Review Request 21846: HIVE-7117: Partitions not inheriting table permissions after alter rename partition

[Ashish Singh]

> On June 3, 2014, 2:37 p.m., Swarnim Kulkarni wrote:
> > common/src/java/org/apache/hadoop/hive/common/FileUtils.java, line 591
> > 
> >
> > Like I said its pretty minor but in my opinion its easy to miss the 
> > intention of a method when someone has to look through half a dozen 
> > arguments to understand that. A simple method name like "renameWithPerms" 
> > make things more clear for a first time consumer.

OK, changed to "renameWithPerms".


- Ashish


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21846/#review44626
---


On June 3, 2014, 9:10 p.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21846/
> ---
> 
> (Updated June 3, 2014, 9:10 p.m.)
> 
> 
> Review request for hive and Szehon Ho.
> 
> 
> Bugs: HIVE-7117
> https://issues.apache.org/jira/browse/HIVE-7117
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Correct FolderPermissionsBase test and change rename method to inherit 
> permissions and/or extended ACLs
> 
> 
> Inherit permissions from parent while renaming directories or files under hive
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 4591678 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java
>  57fe4cf 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> 221b010 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c1790b4 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java 
> ea52537 
> 
> Diff: https://reviews.apache.org/r/21846/diff/
> 
> 
> Testing
> ---
> 
> Tested with itests.
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>

Re: Review Request 21846: HIVE-7117: Partitions not inheriting table permissions after alter rename partition

[Ashish Singh]

> On June 3, 2014, 5:57 a.m., Szehon Ho wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java, line 208
> > 
> >
> > I missed this earlier, can we just return the value returned from here 
> > instead of having an extra returns after this line?

Done!


- Ashish


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21846/#review44602
---


On June 3, 2014, 9:10 p.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21846/
> ---
> 
> (Updated June 3, 2014, 9:10 p.m.)
> 
> 
> Review request for hive and Szehon Ho.
> 
> 
> Bugs: HIVE-7117
> https://issues.apache.org/jira/browse/HIVE-7117
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Correct FolderPermissionsBase test and change rename method to inherit 
> permissions and/or extended ACLs
> 
> 
> Inherit permissions from parent while renaming directories or files under hive
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 4591678 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java
>  57fe4cf 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> 221b010 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c1790b4 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java 
> ea52537 
> 
> Diff: https://reviews.apache.org/r/21846/diff/
> 
> 
> Testing
> ---
> 
> Tested with itests.
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>

Re: Review Request 21846: HIVE-7117: Partitions not inheriting table permissions after alter rename partition

[Ashish Singh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21846/
---

(Updated June 3, 2014, 9:10 p.m.)


Review request for hive and Szehon Ho.


Changes
---

Addressed review comments.


Bugs: HIVE-7117
https://issues.apache.org/jira/browse/HIVE-7117


Repository: hive-git


Description
---

Correct FolderPermissionsBase test and change rename method to inherit 
permissions and/or extended ACLs


Inherit permissions from parent while renaming directories or files under hive


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 4591678 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java
 57fe4cf 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
221b010 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c1790b4 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java 
ea52537 

Diff: https://reviews.apache.org/r/21846/diff/


Testing
---

Tested with itests.


Thanks,

Ashish Singh

[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.

[Jayesh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017134#comment-14017134
 ] 

Jayesh commented on HIVE-7100:
--

{quote}
If your concern was being unable to read the the global 
hive.warehouse.data.skipTrash option value from client code, this seems to be a 
bug that can be fixed. But the patch seems to abandon whatever HIVE-6469 
provides and provide the functionality thru a client API.
{quote}

sorry, my earlier comment about this was bit misleading.
I wanted to say following,

In local metastore mode:
If user set hive.warehouse.data.skipTrash to true from hive client, It also 
updates the configuration on metastore side, ( essentially communicating the 
configuration change)
I believe this is due to metastore running on same JVM as client driver.

In remote metastore mode:
If user set hive.warehouse.data.skipTrash to true from hive client, It does not 
updates the configuration on metastore side,(i.e. it still remains false)
I believe this is due to metastore running on separate JVM, and not 
communicating configuration set by hive client driver which could be right 
behavior by design.

so Yes, The purpose of the patch is to provide a way for client to specify 
whether to skip trash while dropping the table. (by setting configuration 
hive.warehouse.data.skipTrash from hive client driver) . Please also note that 
this configuration can also be set by admin to make required default behavior 
but client ultimately controls the configuration. 

Is this right behavior ?

Thanks
Jay

> Users of hive should be able to specify skipTrash when dropping tables.
> ---
>
> Key: HIVE-7100
> URL: https://issues.apache.org/jira/browse/HIVE-7100
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.13.0
>Reporter: Ravi Prakash
>Assignee: Jayesh
> Attachments: HIVE-7100.patch
>
>
> Users of our clusters are often running up against their quota limits because 
> of Hive tables. When they drop tables, they have to then manually delete the 
> files from HDFS using skipTrash. This is cumbersome and unnecessary. We 
> should enable users to skipTrash directly when dropping tables.
> We should also be able to provide this functionality without polluting SQL 
> syntax.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7170) Fix display_colstats_tbllvl.q in trunk

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017045#comment-14017045
 ] 

Sergey Shelukhin commented on HIVE-7170:


+1

> Fix display_colstats_tbllvl.q in trunk
> --
>
> Key: HIVE-7170
> URL: https://issues.apache.org/jira/browse/HIVE-7170
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Prasanth J
>Assignee: Navis
> Attachments: HIVE-7170.1.patch.txt
>
>
> display_colstats_tbllvl.q test case fails will NPE. Following is the exception
> {code}
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.metastore.StatObjectConverter.fillColumnStatisticsData(StatObjectConverter.java:369)
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.makeColumnStats(MetaStoreDirectSql.java:973)
> at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:888)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:5769)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:5766)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2208)
> at 
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:5766)
> at 
> org.apache.hadoop.hive.metastore.VerifyingObjectStore.getTableColumnStatistics(VerifyingObjectStore.java:103)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.getTableColumnStatistics(Unknown Source)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:3556)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
> at com.sun.proxy.$Proxy11.get_table_statistics_req(Unknown Source)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:1224)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-4867) Deduplicate columns appearing in both the key list and value list of ReduceSinkOperator

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016972#comment-14016972
 ] 

Hive QA commented on HIVE-4867:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12648073/HIVE-4867.3.patch.txt

{color:red}ERROR:{color} -1 due to 58 failed/errored test(s), 5510 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_display_colstats_tbllvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_fs_overwrite
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_louter_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_push_or
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_router_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_25
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input20
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input5
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/379/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/379/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-379/

Messages:
{noformat}
Executing org.apache.hive.ptest.exe

[jira] [Commented] (HIVE-7117) Partitions not inheriting table permissions after alter rename partition

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016570#comment-14016570
 ] 

Hive QA commented on HIVE-7117:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12648065/HIVE-7117.7.patch

{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 5585 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_display_colstats_tbllvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_between_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadata_only_queries
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_schema_evolution
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testDefaults
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabase
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testRenamePartition
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/378/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/378/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-378/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12648065

> Partitions not inheriting table permissions after alter rename partition
> 
>
> Key: HIVE-7117
> URL: https://issues.apache.org/jira/browse/HIVE-7117
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Reporter: Ashish Kumar Singh
>Assignee: Ashish Kumar Singh
> Attachments: HIVE-7117.2.patch, HIVE-7117.3.patch, HIVE-7117.4.patch, 
> HIVE-7117.5.patch, HIVE-7117.6.patch, HIVE-7117.7.patch, HIVE-7117.patch
>
>
> On altering/renaming a partition it must inherit permission of the parent 
> directory, if the flag hive.warehouse.subdir.inherit.perms is set.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21846: HIVE-7117: Partitions not inheriting table permissions after alter rename partition

[Swarnim Kulkarni]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21846/#review44626
---



common/src/java/org/apache/hadoop/hive/common/FileUtils.java


Like I said its pretty minor but in my opinion its easy to miss the 
intention of a method when someone has to look through half a dozen arguments 
to understand that. A simple method name like "renameWithPerms" make things 
more clear for a first time consumer.


- Swarnim Kulkarni


On June 3, 2014, 3:38 a.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21846/
> ---
> 
> (Updated June 3, 2014, 3:38 a.m.)
> 
> 
> Review request for hive and Szehon Ho.
> 
> 
> Bugs: HIVE-7117
> https://issues.apache.org/jira/browse/HIVE-7117
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Correct FolderPermissionsBase test and change rename method to inherit 
> permissions and/or extended ACLs
> 
> 
> Inherit permissions from parent while renaming directories or files under hive
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 4591678 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java
>  57fe4cf 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> 221b010 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c1790b4 
> 
> Diff: https://reviews.apache.org/r/21846/diff/
> 
> 
> Testing
> ---
> 
> Tested with itests.
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>

[jira] [Commented] (HIVE-6394) Implement Timestmap in ParquetSerde

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016417#comment-14016417
 ] 

Hive QA commented on HIVE-6394:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12648057/HIVE-6394.4.patch

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 5514 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_display_colstats_tbllvl
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_parquet_timestamp
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/377/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/377/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-377/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12648057

> Implement Timestmap in ParquetSerde
> ---
>
> Key: HIVE-6394
> URL: https://issues.apache.org/jira/browse/HIVE-6394
> Project: Hive
>  Issue Type: Sub-task
>  Components: Serializers/Deserializers
>Reporter: Jarek Jarcec Cecho
>Assignee: Szehon Ho
>  Labels: Parquet
> Attachments: HIVE-6394.2.patch, HIVE-6394.3.patch, HIVE-6394.4.patch, 
> HIVE-6394.patch
>
>
> This JIRA is to implement timestamp support in Parquet SerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Minimum supported versions for DB backing Metastore

[Lefty Leverenz]

I added a column to the table of supported databases

and updated a table of metastore configuration variables
,
removing some configs that don't exist in any branch and adding some
descriptions and version information.

Reviews, corrections, and further revisions are always welcome.

-- Lefty


On Sat, Apr 19, 2014 at 1:46 AM, Lefty Leverenz 
wrote:

> I found out how MS SQL Server is represented in the patch
> 
>  for HIVE-6927  (Add
> support for MSSQL in schematool), so I guess it is obvious after all:
>
>  public class HiveSchemaHelper {
>>public static final String DB_DERBY = "derby";
>> +  public static final String DB_MSSQL = "mssql";
>>public static final String DB_MYSQL = "mysql";
>>public static final String DB_POSTGRACE = "postgres";
>>public static final String DB_ORACLE = "oracle";
>>
>>
> -- Lefty
>
>
> On Sat, Apr 19, 2014 at 4:25 AM, Lars Francke 
> wrote:
>
>> Sorry for spotting this thread only now.
>>
>> I don't think Postgres 9.1 is required or should be the minimum
>> version considering that RHEL 6 (current version) comes with 8.4 by
>> default. I'd guess that this is one of the most widely deployed
>> versions and lots of clusters are running on this or CentOS 6.
>>
>> On Sat, Apr 19, 2014 at 8:33 AM, Lefty Leverenz 
>> wrote:
>> > Looks good, Ashutosh.  Could you add a column
>> > for javax.jdo.option.ConnectionURL examples?  MySQL is already shown in
>> > previous sections, but it isn't obvious what MS SQL Server should be
>> called
>> > in the parameter value.
>> >
>> > -- Lefty
>> >
>> >
>> > On Fri, Apr 18, 2014 at 5:17 PM, Ashutosh Chauhan > >wrote:
>> >
>> >> I have updated wiki. I decided not to put Derby since I don't think it
>> >> makes sense to recommend Derby for production scenario.
>> >>
>> >>
>> >> On Thu, Apr 17, 2014 at 8:27 PM, Lefty Leverenz <
>> leftylever...@gmail.com
>> >> >wrote:
>> >>
>> >> > +1
>> >> >
>> >> > This seems to be the place for it:
>> >> >
>> >>
>> https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin
>> >> > .
>> >> >
>> >> > I won't volunteer because I've already got my hands full, and
>> besides an
>> >> > expert can do it faster and better.  But I'll find time to edit it.
>> >> >
>> >> > -- Lefty
>> >> >
>> >> >
>> >> > On Thu, Apr 17, 2014 at 9:24 PM, Thejas Nair > >
>> >> > wrote:
>> >> >
>> >> > > +1 Sounds good to me.
>> >> > >
>> >> > > On Thu, Apr 17, 2014 at 2:58 PM, Ashutosh Chauhan <
>> >> hashut...@apache.org>
>> >> > > wrote:
>> >> > > > I don't think we have documented anywhere what versions of mysql
>> /
>> >> > > postgres
>> >> > > > / oracle / ms-sql we are supporting. It will be good to document
>> >> > those. I
>> >> > > > propose following versions:
>> >> > > >
>> >> > > > * Derby -   10.10.1.1  - defined in hive's pom, so
>> all
>> >> unit
>> >> > > > tests run with it.
>> >> > > >
>> >> > > > * MySQL   -   5.6.17   - minimum supported version by
>> >> mysql
>> >> > > > community
>> >> > > >
>> >> > > > * Postgres -  9.1.13- has support for "create
>> table
>> >> if
>> >> > > not
>> >> > > > exists" which is good to have
>> >> > > >
>> >> > > > * Oracle -  11g   - oldest oracle version
>> >> available
>> >> > > to
>> >> > > > download from their site
>> >> > > >
>> >> > > > * MSSQL server -  2008 R2 - one which is currently tested
>> >> against.
>> >> > > >
>> >> > > >
>> >> > > > Thoughts?
>> >> > > >
>> >> > > > Ashutosh
>> >> > >
>> >> > > --
>> >> > > CONFIDENTIALITY NOTICE
>> >> > > NOTICE: This message is intended for the use of the individual or
>> >> entity
>> >> > to
>> >> > > which it is addressed and may contain information that is
>> confidential,
>> >> > > privileged and exempt from disclosure under applicable law. If the
>> >> reader
>> >> > > of this message is not the intended recipient, you are hereby
>> notified
>> >> > that
>> >> > > any printing, copying, dissemination, distribution, disclosure or
>> >> > > forwarding of this communication is strictly prohibited. If you
>> have
>> >> > > received this communication in error, please contact the sender
>> >> > immediately
>> >> > > and delete it from your system. Thank You.
>> >> > >
>> >> >
>> >>
>>
>
>

[jira] [Commented] (HIVE-7168) Don't require to name all columns in analyze statements if stats collection is for all columns

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016314#comment-14016314
 ] 

Hive QA commented on HIVE-7168:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12648048/HIVE-7168.patch

{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 5510 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_display_colstats_tbllvl
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.conf.TestHiveConf.testConfProperties
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testDefaults
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/376/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/376/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-376/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12648048

> Don't require to name all columns in analyze statements if stats collection 
> is for all columns
> --
>
> Key: HIVE-7168
> URL: https://issues.apache.org/jira/browse/HIVE-7168
> Project: Hive
>  Issue Type: Improvement
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-7168.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7173) Support HIVE-4867 on mapjoin of MR Tasks

[Navis (JIRA)]

Navis created HIVE-7173:
---

 Summary: Support HIVE-4867 on mapjoin of MR Tasks
 Key: HIVE-7173
 URL: https://issues.apache.org/jira/browse/HIVE-7173
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor


HIVE-4867 dedups columns in RS for reducer join and RS for order-by. But small 
aliases of mapjoin of MR tasks still contains key columns in value exprs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)