[jira] [Commented] (HIVE-4373) Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643570#comment-13643570
 ] 

Carl Steinbach commented on HIVE-4373:
--

I left some comments on reviewboard. Thanks.

> Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is 
> incorrect
> 
>
> Key: HIVE-4373
> URL: https://issues.apache.org/jira/browse/HIVE-4373
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.11.0
>Reporter: Deepesh Khandelwal
>Assignee: Thejas M Nair
>Priority: Minor
> Fix For: 0.11.0
>
> Attachments: HIVE-4373.1.patch
>
>
> When running beeline
> {code}
> % beeline -u 'jdbc:hive2://localhost:1' -n hive -p passwd -d 
> org.apache.hive.jdbc.HiveDriver
> Connecting to jdbc:hive2://localhost:1
> Connected to: Hive (version 0.10.0)
> Driver: Hive (version 0.11.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> {code}
> The Hive version in the "Connected to: " string says 0.10.0 instead of 0.11.0.
> Looking at the code it seems that the version is hardcoded at two places:
> line 250 in jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java
> line 833 in jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4433) Fix C++ Thrift bindings broken in HIVE-4322

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643572#comment-13643572
 ] 

Carl Steinbach commented on HIVE-4433:
--

You can reproduce the error by running the following command:

% ant compile-cpp -Dthrift.home=$THRIFT_HOME

Here's the error message I get:

{noformat}
 [exec] g++ -m64 -DARCH64 -dynamiclib 
/Users/carl/Work/repos/hive-test/build/metastore/objs/ThriftHiveMetastore.o 
/Users/carl/Work/repos/hive-test/build/metastore/objs/hive_metastore_constants.o
 /Users/carl/Work/repos/hive-test/build/metastore/objs/hive_metastore_types.o 
/Users/carl/Work/repos/hive-test/build/service/objs/ThriftHive.o 
/Users/carl/Work/repos/hive-test/build/service/objs/hive_service_constants.o 
/Users/carl/Work/repos/hive-test/build/service/objs/hive_service_types.o 
/Users/carl/Work/repos/hive-test/build/ql/objs/queryplan_types.o 
/Users/carl/Work/repos/hive-test/build/ql/objs/queryplan_constants.o 
/Users/carl/Work/repos/hive-test/build/odbc/objs/hiveclient.o 
/Users/carl/Work/repos/hive-test/build/odbc/objs/HiveResultSet.o 
/Users/carl/Work/repos/hive-test/build/odbc/objs/HiveColumnDesc.o 
/Users/carl/Work/repos/hive-test/build/odbc/objs/HiveRowSet.o 
/Users/carl/Work/repos/hive-test/build/odbc/objs/hiveclienthelper.o 
-L/usr/local/lib -lthrift -L/usr/local/lib -lfb303 -o 
/Users/carl/Work/repos/hive-test/build/odbc/lib/libhiveclient.so.1.0.0 \
 [exec] && ln -sf libhiveclient.so.1.0.0 
/Users/carl/Work/repos/hive-test/build/odbc/lib/libhiveclient.so
 [exec] Undefined symbols for architecture x86_64:
 [exec]   
"Apache::Hadoop::Hive::SkewedValueList::operator<(Apache::Hadoop::Hive::SkewedValueList
 const&) const", referenced from:
 [exec]   
std::less::operator()(Apache::Hadoop::Hive::SkewedValueList
 const&, Apache::Hadoop::Hive::SkewedValueList const&) constin 
hive_metastore_types.o
 [exec] ld: symbol(s) not found for architecture x86_64
 [exec] collect2: ld returned 1 exit status
 [exec] make: *** 
[/Users/carl/Work/repos/hive-test/build/odbc/lib/libhiveclient.so.1.0.0] Error 1

BUILD FAILED
/Users/carl/Work/repos/hive-test/build.xml:242: The following error occurred 
while executing this line:
/Users/carl/Work/repos/hive-test/build.xml:109: The following error occurred 
while executing this line:
/Users/carl/Work/repos/hive-test/build.xml:111: The following error occurred 
while executing this line:
/Users/carl/Work/repos/hive-test/odbc/build.xml:69: exec returned: 2
{noformat}

The problem is that STL maps require the '<' operator to be defined for the key 
type, and SkewedValueList doesn't satisfy that requirement.

> Fix C++ Thrift bindings broken in HIVE-4322
> ---
>
> Key: HIVE-4433
> URL: https://issues.apache.org/jira/browse/HIVE-4433
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Thrift API
>Affects Versions: 0.11.0
>Reporter: Carl Steinbach
>Assignee: Samuel Yuan
>Priority: Blocker
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643606#comment-13643606
 ] 

Hudson commented on HIVE-4424:
--

Integrated in Hive-trunk-hadoop2 #176 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop2/176/])
HIVE-4424 MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
(Namit Jain) (Revision 1476119)

 Result = FAILURE
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476119
Files : 
* 
/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java.orig


> MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
> ---
>
> Key: HIVE-4424
> URL: https://issues.apache.org/jira/browse/HIVE-4424
> Project: Hive
>  Issue Type: Bug
>Reporter: Namit Jain
>Assignee: Namit Jain
> Fix For: 0.12.0
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3708) Add mapreduce workflow information to job configuration

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643607#comment-13643607
 ] 

Hudson commented on HIVE-3708:
--

Integrated in Hive-trunk-hadoop2 #176 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop2/176/])
HIVE-3708 : Add mapreduce workflow information to job configuration (Billie 
Rinaldi via Ashutosh Chauhan) (Revision 1476252)

 Result = FAILURE
hashutosh : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476252
Files : 
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java


> Add mapreduce workflow information to job configuration
> ---
>
> Key: HIVE-3708
> URL: https://issues.apache.org/jira/browse/HIVE-3708
> Project: Hive
>  Issue Type: Improvement
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Fix For: 0.12.0
>
> Attachments: HIVE-3708_1.patch, HIVE-3708.patch, HIVE-3708.patch
>
>
> Adding workflow properties to the job configuration would enable logging and 
> analysis of workflows in addition to individual MapReduce jobs.  Suggested 
> properties include a workflow ID, workflow name, adjacency list connecting 
> nodes in the workflow, and the name of the current node in the workflow.
> mapreduce.workflow.id - a unique ID for the workflow, ideally prepended with 
> the application name
> e.g. hive_
> mapreduce.workflow.name - a name for the workflow, to distinguish this 
> workflow from other workflows and to group different runs of the same workflow
> e.g. hive query string
> mapreduce.workflow.adjacency - an adjacency list for the workflow graph, 
> encoded as mapreduce.workflow.adjacency. =  of target nodes>
> mapreduce.workflow.node.name - the name of the node corresponding to this 
> MapReduce job in the workflow adjacency list

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4409) Prevent incompatible column type changes

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643605#comment-13643605
 ] 

Hudson commented on HIVE-4409:
--

Integrated in Hive-trunk-hadoop2 #176 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop2/176/])
HIVE-4424 MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
(Namit Jain) (Revision 1476119)

 Result = FAILURE
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476119
Files : 
* 
/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java.orig


> Prevent incompatible column type changes
> 
>
> Key: HIVE-4409
> URL: https://issues.apache.org/jira/browse/HIVE-4409
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI, Metastore
>Affects Versions: 0.10.0
>Reporter: Dilip Joseph
>Assignee: Dilip Joseph
>Priority: Minor
> Fix For: 0.12.0
>
> Attachments: hive.4409.1.patch, HIVE-4409.D10539.1.patch, 
> HIVE-4409.D10539.2.patch
>
>
> If a user changes the type of an existing column of a partitioned table to an 
> incompatible type, subsequent accesses of old partitions will result in a 
> ClassCastException (see example below).  We should prevent the user from 
> making incompatible type changes.  This feature will be controlled by a new 
> config parameter.
> Example:
> CREATE TABLE test_table123 (a INT, b MAP) PARTITIONED BY (ds 
> STRING) STORED AS SEQUENCEFILE;
> INSERT OVERWRITE TABLE test_table123 PARTITION(ds="foo1") SELECT 1, MAP("a1", 
> "b1") FROM src LIMIT 1;
> SELECT * from test_table123 WHERE ds="foo1";
> ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING);
> SELECT * from test_table123 WHERE ds="foo1";
> The last SELECT fails with the following exception:
> Failed with exception java.io.IOException:java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
> cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> java.io.IOException: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector 
> cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
>   at 
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
>   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
>   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406)
>   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
>   at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790)
>   at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108)
> The ALTER TABLE statement is blocked if you set the following parameter, 
> introduced int the fix to this JIRA:
> SET hive.metastore.disallow.incompatible.col.type.changes=true;

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: HIVE-4373 - Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect

[Thejas Nair]

> On April 27, 2013, 6:59 a.m., Carl Steinbach wrote:
> > common/src/scripts/saveVersion.sh, line 66
> > 
> >
> > I think the output directory was correct to begin with. Please revert.

HiveVersionInfo is in org/apache/hive/common/util/, so this package-info.java 
with the annotations expected by HiveVersionInfo needs to be in same package. 
Otherwise  HiveVersionInfo will not work.
Hence the change to move it to this dir. Please let me know if you disagree. 


> On April 27, 2013, 6:59 a.m., Carl Steinbach wrote:
> > common/build.xml, line 29
> > 
> >
> > This extra property is only referenced in one place. I think it would 
> > be cleaner to remove it.

Changing that in new patch.


- Thejas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10812/#review19822
---


On April 27, 2013, 2:53 a.m., Thejas Nair wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10812/
> ---
> 
> (Updated April 27, 2013, 2:53 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Description
> ---
> 
> HIVE-4373
> 
> 
> This addresses bug HIVE-4373.
> https://issues.apache.org/jira/browse/HIVE-4373
> 
> 
> Diffs
> -
> 
>   common/build.xml 731f26e 
>   common/src/scripts/saveVersion.sh a917d4a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java 3d1a969 
>   jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java 938dd9a 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 18594cb 
> 
> Diff: https://reviews.apache.org/r/10812/diff/
> 
> 
> Testing
> ---
> 
> Unit test included. Also tested with remote HS2 server.
> 
> 
> Thanks,
> 
> Thejas Nair
> 
>

Re: Review Request: HIVE-4373 - Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10812/
---

(Updated April 27, 2013, 9:58 a.m.)


Review request for hive.


Changes
---

Patch addressing Carl's review comments.


Description
---

HIVE-4373


This addresses bug HIVE-4373.
https://issues.apache.org/jira/browse/HIVE-4373


Diffs (updated)
-

  common/build.xml 731f26e 
  common/src/scripts/saveVersion.sh a917d4a 
  jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java 3d1a969 
  jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java 938dd9a 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
18594cb 

Diff: https://reviews.apache.org/r/10812/diff/


Testing
---

Unit test included. Also tested with remote HS2 server.


Thanks,

Thejas Nair

[jira] [Updated] (HIVE-4373) Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-4373:


Attachment: HIVE-4373.2.patch

New patch addressing review comments from [~cwsteinbach]

> Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is 
> incorrect
> 
>
> Key: HIVE-4373
> URL: https://issues.apache.org/jira/browse/HIVE-4373
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.11.0
>Reporter: Deepesh Khandelwal
>Assignee: Thejas M Nair
>Priority: Minor
> Fix For: 0.11.0
>
> Attachments: HIVE-4373.1.patch, HIVE-4373.2.patch
>
>
> When running beeline
> {code}
> % beeline -u 'jdbc:hive2://localhost:1' -n hive -p passwd -d 
> org.apache.hive.jdbc.HiveDriver
> Connecting to jdbc:hive2://localhost:1
> Connected to: Hive (version 0.10.0)
> Driver: Hive (version 0.11.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> {code}
> The Hive version in the "Connected to: " string says 0.10.0 instead of 0.11.0.
> Looking at the code it seems that the version is hardcoded at two places:
> line 250 in jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java
> line 833 in jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4434) some tables was created but no SDS, and can't drop these tables

[gongguan.zhang (JIRA)]

gongguan.zhang created HIVE-4434:


 Summary: some tables was created but no SDS, and can't drop these 
tables
 Key: HIVE-4434
 URL: https://issues.apache.org/jira/browse/HIVE-4434
 Project: Hive
  Issue Type: Bug
  Components: CLI
Affects Versions: 0.8.1, 0.8.0
 Environment: OS:Linux x86_64
hive 0.8.0
hadoop-0.21
Reporter: gongguan.zhang


all these tables are invalid,when selecting,get the following exception:
hive> select *from tmp_visitor_vip_access_distinct_2_20130401 limit 1;
FAILED: Hive Internal Error: java.lang.NullPointerException(null)
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:859)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:787)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:875)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7120)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:376)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:334)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:843)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:593)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

when dropping table,get the next following exceptions:
hive> drop table tmp_visitor_vip_access_distinct_2_20130401;
FAILED: Hive Internal Error: java.lang.NullPointerException(null)
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:859)
at 
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeDropTable(DDLSemanticAnalyzer.java:693)
at 
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:203)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:376)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:334)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:843)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:593)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

and the metadata info in mysql is:
TBL_ID: 43832
   CREATE_TIME: 1367017748
 DB_ID: 1
  LAST_ACCESS_TIME: 0
 OWNER: acorn
 RETENTION: 0
 SD_ID: NULL
  TBL_NAME: tmp_visitor_vip_access_distinct_2_20130401
  TBL_TYPE: MANAGED_TABLE
VIEW_EXPANDED_TEXT: NULL
VIEW_ORIGINAL_TEXT: NULL
 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4434) some tables was created but no SDS, and can't drop these tables

[gongguan.zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643621#comment-13643621
 ] 

gongguan.zhang commented on HIVE-4434:
--

tTable.getSd() is null, and there is files in 
hdfs://***/warehouse/tmp_visitor_vip_access_distinct_2_20130401


> some tables was created but no SDS, and can't drop these tables
> ---
>
> Key: HIVE-4434
> URL: https://issues.apache.org/jira/browse/HIVE-4434
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.8.0, 0.8.1
> Environment: OS:Linux x86_64
> hive 0.8.0
> hadoop-0.21
>Reporter: gongguan.zhang
>
> all these tables are invalid,when selecting,get the following exception:
> hive> select *from tmp_visitor_vip_access_distinct_2_20130401 limit 1;
> FAILED: Hive Internal Error: java.lang.NullPointerException(null)
> java.lang.NullPointerException
>   at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:859)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:787)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:875)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7120)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:376)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:334)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:843)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:593)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
> when dropping table,get the next following exceptions:
> hive> drop table tmp_visitor_vip_access_distinct_2_20130401;
> FAILED: Hive Internal Error: java.lang.NullPointerException(null)
> java.lang.NullPointerException
>   at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:859)
>   at 
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeDropTable(DDLSemanticAnalyzer.java:693)
>   at 
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:203)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:376)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:334)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:843)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:593)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
> and the metadata info in mysql is:
> TBL_ID: 43832
>CREATE_TIME: 1367017748
>  DB_ID: 1
>   LAST_ACCESS_TIME: 0
>  OWNER: acorn
>  RETENTION: 0
>  SD_ID: NULL
>   TBL_NAME: tmp_visitor_vip_access_distinct_2_20130401
>   TBL_TYPE: MANAGED_TABLE
> VIEW_EXPANDED_TEXT: NULL
> VIEW_ORIGINAL_TEXT: NULL
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4019) Ability to create and drop temporary partition function

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643649#comment-13643649
 ] 

Brock Noland commented on HIVE-4019:


[~le...@hortonworks.com] Nice work on the docs!

[~rhbutani] Just checking, was the +1 in response to the patch or Lefty's 
document comment?

> Ability to create and drop temporary partition function
> ---
>
> Key: HIVE-4019
> URL: https://issues.apache.org/jira/browse/HIVE-4019
> Project: Hive
>  Issue Type: New Feature
>  Components: PTF-Windowing
>Reporter: Ashutosh Chauhan
>Assignee: Brock Noland
> Attachments: HIVE-4019-1.patch, HIVE-4019.2.patch, HIVE-4019-3.patch, 
> HIVE-4019-4.patch, HIVE-4019-5.patch, HIVE-4019-6.patch, HIVE-4019-7.patch, 
> hive-4019.q
>
>
> Just like udf/udaf/udtf functions, user should be able to add and drop custom 
> partitioning functions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4019) Ability to create and drop temporary partition function

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643701#comment-13643701
 ] 

Harish Butani commented on HIVE-4019:
-

+1 for the patch

> Ability to create and drop temporary partition function
> ---
>
> Key: HIVE-4019
> URL: https://issues.apache.org/jira/browse/HIVE-4019
> Project: Hive
>  Issue Type: New Feature
>  Components: PTF-Windowing
>Reporter: Ashutosh Chauhan
>Assignee: Brock Noland
> Attachments: HIVE-4019-1.patch, HIVE-4019.2.patch, HIVE-4019-3.patch, 
> HIVE-4019-4.patch, HIVE-4019-5.patch, HIVE-4019-6.patch, HIVE-4019-7.patch, 
> hive-4019.q
>
>
> Just like udf/udaf/udtf functions, user should be able to add and drop custom 
> partitioning functions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643704#comment-13643704
 ] 

Hudson commented on HIVE-4068:
--

Integrated in Hive-trunk-hadoop2 #177 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop2/177/])
HIVE-4068 Size of aggregation buffer which uses non-primitive type is not 
estimated correctly (Navis) (Revision 1476483)

 Result = FAILURE
navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476483
Files : 
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBridge.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java


> Size of aggregation buffer which uses non-primitive type is not estimated 
> correctly
> ---
>
> Key: HIVE-4068
> URL: https://issues.apache.org/jira/browse/HIVE-4068
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.12.0
>
> Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, 
> HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch
>
>
> Currently, hive assumes an aggregation buffer which holds a map is occupying 
> just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can 
> be thrown (especially for >1k buffer). 
> workaround : set hive.map.aggr.hash.percentmemory= default(0.5)>

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4420) HCatalog unit tests stop after a failure

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643705#comment-13643705
 ] 

Hudson commented on HIVE-4420:
--

Integrated in Hive-trunk-hadoop2 #177 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop2/177/])
HIVE-4420 : HCatalog unit tests stop after a failure (Alan Gates via 
Sushanth Sowmyan) (Revision 1476444)

 Result = FAILURE
khorgath : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476444
Files : 
* /hive/trunk/hcatalog/build.xml


> HCatalog unit tests stop after a failure
> 
>
> Key: HIVE-4420
> URL: https://issues.apache.org/jira/browse/HIVE-4420
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure, HCatalog
>Affects Versions: 0.11.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.12.0
>
> Attachments: HIVE-4420.patch
>
>
> HCatalog splits it's different components (core, hcatalog-pig-adaptor, 
> webhcat, etc.) into separate build.xml files, all controlled by the top 
> build.xml.  A side effect of this is that when the user does "ant test" and 
> the top build file is iterating through the tests, if one of the component's 
> test targets fail then unit tests are not run for subsequent components.  
> This means a failure in one component can mask issues in others.  
> test should run on all the components even when there is a failure in one of 
> the components.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-hadoop2 - Build # 177 - Still Failing

[Apache Jenkins Server]

Changes for Build #143

Changes for Build #144
[hashutosh] HIVE-3908 : create view statement's outputs contains the view and a 
temporary dir. (Prasad Mujumdar via Ashutosh Chauhan)

[hashutosh] HIVE-4302 : Fix how RowSchema and RowResolver are set on 
ReduceSinkOp that precedes PTFOp (Harish Butani via Ashutosh Chauhan)


Changes for Build #145
[hashutosh] HIVE-3992 : Hive RCFile::sync(long) does a sub-sequence linear 
search for sync blocks (Gopal V via Ashutosh Chauhan)


Changes for Build #146

Changes for Build #147
[hashutosh] HIVE-2340 : optimize orderby followed by a groupby (Navis via 
Ashutosh Chauhan)


Changes for Build #148
[navis] Missing test results from HIVE-1953 (Vikram Dixit K via Navis)

[namit] HIVE-4314 Result of mapjoin_test_outer.q is not deterministic
(Navis via namit)

[navis] HIVE-1953 Hive should process comments in CliDriver (Vikram Dixit K via 
Navis)

[navis] HIVE-3308 Mixing avro and snappy gives null values (Bennie Schut via 
Navis)

[hashutosh] HIVE-4311 : DOS line endings in auto_join26.q (Gunther Hagleitner 
via Ashutosh Chauhan)


Changes for Build #149
[hashutosh] HIVE-4107 : Update Hive 0.10.0 RELEASE_NOTES.txt (Thejas Nair via 
Ashutosh Chauhan)

[hashutosh] HIVE-4271 : Limit precision of decimal type (Gunther Hagleitner via 
Ashutosh Chauhan)

[hashutosh] HIVE-4319 : Revert changes checked-in as part of 1953 (Vikram Dixit 
via Ashutosh Chauhan)

[khorgath] HCATALOG-632 Fixing ORC File usage with HCatalog


Changes for Build #150
[hashutosh] HIVE-4325 : Merge HCat NOTICE file with Hive NOTICE file (Alan 
Gates via Ashutosh Chauhan)

[gates] Moving CHANGES.txt from under trunk to hcatalog-historical

[gates] HIVE-4236 - Clean up remaining items in hive/hcatalog/historical/trunk 
(gates)


Changes for Build #151
[hashutosh] HIVE-4078 : Delay the serialize-deserialize pair in 
CommonJoinTaskDispatcher (Gopal V via Ashutosh Chauhan)

[gangtimliu] HIVE-4337: Update list bucketing test results (Samuel Yuan via 
Gang Tim Liu)

[hashutosh] HIVE-4306 : PTFDeserializer should reconstruct OIs based on InputOI 
passed to PTFOperator (Harish Butani and Prajakta Kalmegh via Ashutosh Chauhan)

[hashutosh] HIVE-4334 : ctas test on hadoop 2 has outdated golden file (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4332 : orc*.q tests fail on hadoop 2 (Gunther Hagleitner via 
Ashutosh Chauhan)

[hashutosh] HIVE-4120 : Implement decimal encoding for ORC (Owen Omalley via 
Ashutosh Chauhan)

[hashutosh] HIVE-4303 : 2 empty java files in hcatalog (Owen Omalley via 
Ashutosh Chauhan)

[hashutosh] HIVE-4335 : serde_regex test fails on hadoop 2 (Gunther Hagleitner 
via Ashutosh Chauhan)


Changes for Build #152
[namit] HIVE-4328 Change default bigtable selection policy for sort-merge joins
(Namit Jain via Ashutosh Chauhan)

[hashutosh] Preparing for 0.12 development

[hashutosh] HIVE-4323 : sqlline dependency is not required (Thejas Nair via 
Ashutosh Chauhan)

[namit] HIVE-4336 Selecting from a view, and another view that also selects 
from that view fails
(Kevin Wilfong via namit)


Changes for Build #153
[namit] HIVE-4241 optimize hive.enforce.sorting and hive.enforce bucketing join
(Namit Jain via Gang Tim Liu)


Changes for Build #154
[namit] HIVE-4294 Single sourced multi query cannot handle lateral view
(Navis via namit)


Changes for Build #155

Changes for Build #156
[hashutosh] HIVE-4339 : build fails after branch (hcatalog version not updated) 
(Gunther Hagleitner via Ashutosh Chauhan)


Changes for Build #157
[hashutosh] HIVE-4296 : ant thriftif  fails on  hcatalog (Roshan Naik via 
Ashutosh Chauhan)

[gangtimliu] HIVE-4322: SkewedInfo in Metastore Thrift API cannot be 
deserialized in Python (Samuel Yuan via Gang Tim Liu)


Changes for Build #158

Changes for Build #159

Changes for Build #160
[navis] HIVE-3179 HBase Handler doesn't handle NULLs properly (Lars Francke via 
Navis)


Changes for Build #161
[cws] HIVE-4268. Beeline should support the -f option (Rob Weltman via cws)


Changes for Build #162
[hashutosh] HIVE-4275 : Hive does not differentiate scheme and authority in 
file uris (Vikram Dixit via Ashutosh Chauhan)

[hashutosh] HIVE-4352 : Guava not getting included in build package (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4315 : enable doAs in unsecure mode for hive server2, when MR 
job runs locally (Thejas Nair via Ashutosh Chauhan)

[hashutosh] HIVE-4105 : Hive MapJoinOperator unnecessarily deserializes values 
for all join-keys (Vinod KV via Ashutosh Chauhan)

[hashutosh] HIVE-4327 : NPE in constant folding with decimal (Gunther 
Hagleitner via Ashutosh Chauhan)

[namit] HIVE-4167 Hive converts bucket map join to SMB join even when tables 
are not sorted
(Namit Jain via Ashutosh)

[namit] HIVE-4308 Newly added test TestCliDriver.hiveprofiler_union0 is failing 
on trunk
(Navis via namit)

[namit] HIVE-3996 Correctly enforce the memory limit on the multi-table map-join
(Vikram Dixit via namit)

[navis] HIVE-4261 union_remove_10 is fail

[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643742#comment-13643742
 ] 

Phabricator commented on HIVE-3682:
---

ashutoshc has accepted the revision "HIVE-3682 [jira] when output hive table to 
file,users should could have a separator of their own choice".

  +1 will commit if tests pass.

REVISION DETAIL
  https://reviews.facebook.net/D10275

BRANCH
  HIVE-3682

ARCANIST PROJECT
  hive

To: JIRA, ashutoshc, omalley, khorgath


> when output hive table to file,users should could have a separator of their 
> own choice
> --
>
> Key: HIVE-3682
> URL: https://issues.apache.org/jira/browse/HIVE-3682
> Project: Hive
>  Issue Type: New Feature
>  Components: CLI
>Affects Versions: 0.8.1
> Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 
> 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux
> java version "1.6.0_25"
> hadoop-0.20.2-cdh3u0
> hive-0.8.1
>Reporter: caofangkun
>Assignee: Sushanth Sowmyan
> Fix For: 0.11.0
>
> Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, 
> HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.with.serde.patch
>
>
> By default,when output hive table to file ,columns of the Hive table are 
> separated by ^A character (that is \001).
> But indeed users should have the right to set a seperator of their own choice.
> Usage Example:
> create table for_test (key string, value string);
> load data local inpath './in1.txt' into table for_test
> select * from for_test;
> UT-01：default separator is \001 line separator is \n
> insert overwrite local directory './test-01' 
> select * from src ;
> create table array_table (a array, b array)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> COLLECTION ITEMS TERMINATED BY ',';
> load data local inpath "../hive/examples/files/arraytest.txt" overwrite into 
> table table2;
> CREATE TABLE map_table (foo STRING , bar MAP)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> COLLECTION ITEMS TERMINATED BY ','
> MAP KEYS TERMINATED BY ':'
> STORED AS TEXTFILE;
> UT-02：defined field separator as ':'
> insert overwrite local directory './test-02' 
> row format delimited 
> FIELDS TERMINATED BY ':' 
> select * from src ;
> UT-03: line separator DO NOT ALLOWED to define as other separator 
> insert overwrite local directory './test-03' 
> row format delimited 
> FIELDS TERMINATED BY ':' 
> select * from src ;
> UT-04: define map separators 
> insert overwrite local directory './test-04' 
> row format delimited 
> FIELDS TERMINATED BY '\t'
> COLLECTION ITEMS TERMINATED BY ','
> MAP KEYS TERMINATED BY ':'
> select * from src;

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-hadoop2 - Build # 178 - Still Failing

[Apache Jenkins Server]

Changes for Build #143

Changes for Build #144
[hashutosh] HIVE-3908 : create view statement's outputs contains the view and a 
temporary dir. (Prasad Mujumdar via Ashutosh Chauhan)

[hashutosh] HIVE-4302 : Fix how RowSchema and RowResolver are set on 
ReduceSinkOp that precedes PTFOp (Harish Butani via Ashutosh Chauhan)


Changes for Build #145
[hashutosh] HIVE-3992 : Hive RCFile::sync(long) does a sub-sequence linear 
search for sync blocks (Gopal V via Ashutosh Chauhan)


Changes for Build #146

Changes for Build #147
[hashutosh] HIVE-2340 : optimize orderby followed by a groupby (Navis via 
Ashutosh Chauhan)


Changes for Build #148
[navis] Missing test results from HIVE-1953 (Vikram Dixit K via Navis)

[namit] HIVE-4314 Result of mapjoin_test_outer.q is not deterministic
(Navis via namit)

[navis] HIVE-1953 Hive should process comments in CliDriver (Vikram Dixit K via 
Navis)

[navis] HIVE-3308 Mixing avro and snappy gives null values (Bennie Schut via 
Navis)

[hashutosh] HIVE-4311 : DOS line endings in auto_join26.q (Gunther Hagleitner 
via Ashutosh Chauhan)


Changes for Build #149
[hashutosh] HIVE-4107 : Update Hive 0.10.0 RELEASE_NOTES.txt (Thejas Nair via 
Ashutosh Chauhan)

[hashutosh] HIVE-4271 : Limit precision of decimal type (Gunther Hagleitner via 
Ashutosh Chauhan)

[hashutosh] HIVE-4319 : Revert changes checked-in as part of 1953 (Vikram Dixit 
via Ashutosh Chauhan)

[khorgath] HCATALOG-632 Fixing ORC File usage with HCatalog


Changes for Build #150
[hashutosh] HIVE-4325 : Merge HCat NOTICE file with Hive NOTICE file (Alan 
Gates via Ashutosh Chauhan)

[gates] Moving CHANGES.txt from under trunk to hcatalog-historical

[gates] HIVE-4236 - Clean up remaining items in hive/hcatalog/historical/trunk 
(gates)


Changes for Build #151
[hashutosh] HIVE-4078 : Delay the serialize-deserialize pair in 
CommonJoinTaskDispatcher (Gopal V via Ashutosh Chauhan)

[gangtimliu] HIVE-4337: Update list bucketing test results (Samuel Yuan via 
Gang Tim Liu)

[hashutosh] HIVE-4306 : PTFDeserializer should reconstruct OIs based on InputOI 
passed to PTFOperator (Harish Butani and Prajakta Kalmegh via Ashutosh Chauhan)

[hashutosh] HIVE-4334 : ctas test on hadoop 2 has outdated golden file (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4332 : orc*.q tests fail on hadoop 2 (Gunther Hagleitner via 
Ashutosh Chauhan)

[hashutosh] HIVE-4120 : Implement decimal encoding for ORC (Owen Omalley via 
Ashutosh Chauhan)

[hashutosh] HIVE-4303 : 2 empty java files in hcatalog (Owen Omalley via 
Ashutosh Chauhan)

[hashutosh] HIVE-4335 : serde_regex test fails on hadoop 2 (Gunther Hagleitner 
via Ashutosh Chauhan)


Changes for Build #152
[namit] HIVE-4328 Change default bigtable selection policy for sort-merge joins
(Namit Jain via Ashutosh Chauhan)

[hashutosh] Preparing for 0.12 development

[hashutosh] HIVE-4323 : sqlline dependency is not required (Thejas Nair via 
Ashutosh Chauhan)

[namit] HIVE-4336 Selecting from a view, and another view that also selects 
from that view fails
(Kevin Wilfong via namit)


Changes for Build #153
[namit] HIVE-4241 optimize hive.enforce.sorting and hive.enforce bucketing join
(Namit Jain via Gang Tim Liu)


Changes for Build #154
[namit] HIVE-4294 Single sourced multi query cannot handle lateral view
(Navis via namit)


Changes for Build #155

Changes for Build #156
[hashutosh] HIVE-4339 : build fails after branch (hcatalog version not updated) 
(Gunther Hagleitner via Ashutosh Chauhan)


Changes for Build #157
[hashutosh] HIVE-4296 : ant thriftif  fails on  hcatalog (Roshan Naik via 
Ashutosh Chauhan)

[gangtimliu] HIVE-4322: SkewedInfo in Metastore Thrift API cannot be 
deserialized in Python (Samuel Yuan via Gang Tim Liu)


Changes for Build #158

Changes for Build #159

Changes for Build #160
[navis] HIVE-3179 HBase Handler doesn't handle NULLs properly (Lars Francke via 
Navis)


Changes for Build #161
[cws] HIVE-4268. Beeline should support the -f option (Rob Weltman via cws)


Changes for Build #162
[hashutosh] HIVE-4275 : Hive does not differentiate scheme and authority in 
file uris (Vikram Dixit via Ashutosh Chauhan)

[hashutosh] HIVE-4352 : Guava not getting included in build package (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4315 : enable doAs in unsecure mode for hive server2, when MR 
job runs locally (Thejas Nair via Ashutosh Chauhan)

[hashutosh] HIVE-4105 : Hive MapJoinOperator unnecessarily deserializes values 
for all join-keys (Vinod KV via Ashutosh Chauhan)

[hashutosh] HIVE-4327 : NPE in constant folding with decimal (Gunther 
Hagleitner via Ashutosh Chauhan)

[namit] HIVE-4167 Hive converts bucket map join to SMB join even when tables 
are not sorted
(Namit Jain via Ashutosh)

[namit] HIVE-4308 Newly added test TestCliDriver.hiveprofiler_union0 is failing 
on trunk
(Navis via namit)

[namit] HIVE-3996 Correctly enforce the memory limit on the multi-table map-join
(Vikram Dixit via namit)

[navis] HIVE-4261 union_remove_10 is fail

Hive-trunk-h0.21 - Build # 2081 - Still Failing

[Apache Jenkins Server]

Changes for Build #2049
[hashutosh] HIVE-3985 : Update new UDAFs introduced for Windowing to work with 
new Decimal Type (Brock Noland via Ashutosh Chauhan)

[hashutosh] HIVE-3840 : hive cli null representation in output is inconsistent 
(Thejas Nair via Ashutosh Chauhan)

[hashutosh] HIVE-4262 : fix last_value UDAF behavior (Harish Butani via 
Ashutosh Chauhan)

[hashutosh] HIVE-4292 : hiveserver2 should support -hiveconf commandline 
parameter (Thejas Nair via Ashutosh Chauhan)


Changes for Build #2050
[hashutosh] HIVE-3908 : create view statement's outputs contains the view and a 
temporary dir. (Prasad Mujumdar via Ashutosh Chauhan)

[hashutosh] HIVE-4302 : Fix how RowSchema and RowResolver are set on 
ReduceSinkOp that precedes PTFOp (Harish Butani via Ashutosh Chauhan)


Changes for Build #2051
[hashutosh] HIVE-3992 : Hive RCFile::sync(long) does a sub-sequence linear 
search for sync blocks (Gopal V via Ashutosh Chauhan)


Changes for Build #2052

Changes for Build #2053
[navis] Missing test results from HIVE-1953 (Vikram Dixit K via Navis)

[namit] HIVE-4314 Result of mapjoin_test_outer.q is not deterministic
(Navis via namit)

[navis] HIVE-1953 Hive should process comments in CliDriver (Vikram Dixit K via 
Navis)

[navis] HIVE-3308 Mixing avro and snappy gives null values (Bennie Schut via 
Navis)

[hashutosh] HIVE-4311 : DOS line endings in auto_join26.q (Gunther Hagleitner 
via Ashutosh Chauhan)

[hashutosh] HIVE-2340 : optimize orderby followed by a groupby (Navis via 
Ashutosh Chauhan)


Changes for Build #2054
[khorgath] HCATALOG-632 Fixing ORC File usage with HCatalog


Changes for Build #2055
[hashutosh] HIVE-4107 : Update Hive 0.10.0 RELEASE_NOTES.txt (Thejas Nair via 
Ashutosh Chauhan)

[hashutosh] HIVE-4271 : Limit precision of decimal type (Gunther Hagleitner via 
Ashutosh Chauhan)

[hashutosh] HIVE-4319 : Revert changes checked-in as part of 1953 (Vikram Dixit 
via Ashutosh Chauhan)


Changes for Build #2056
[hashutosh] HIVE-4078 : Delay the serialize-deserialize pair in 
CommonJoinTaskDispatcher (Gopal V via Ashutosh Chauhan)

[gangtimliu] HIVE-4337: Update list bucketing test results (Samuel Yuan via 
Gang Tim Liu)

[hashutosh] HIVE-4306 : PTFDeserializer should reconstruct OIs based on InputOI 
passed to PTFOperator (Harish Butani and Prajakta Kalmegh via Ashutosh Chauhan)

[hashutosh] HIVE-4334 : ctas test on hadoop 2 has outdated golden file (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4332 : orc*.q tests fail on hadoop 2 (Gunther Hagleitner via 
Ashutosh Chauhan)

[hashutosh] HIVE-4120 : Implement decimal encoding for ORC (Owen Omalley via 
Ashutosh Chauhan)

[hashutosh] HIVE-4303 : 2 empty java files in hcatalog (Owen Omalley via 
Ashutosh Chauhan)

[hashutosh] HIVE-4335 : serde_regex test fails on hadoop 2 (Gunther Hagleitner 
via Ashutosh Chauhan)

[hashutosh] HIVE-4325 : Merge HCat NOTICE file with Hive NOTICE file (Alan 
Gates via Ashutosh Chauhan)

[gates] Moving CHANGES.txt from under trunk to hcatalog-historical

[gates] HIVE-4236 - Clean up remaining items in hive/hcatalog/historical/trunk 
(gates)


Changes for Build #2057
[namit] HIVE-4328 Change default bigtable selection policy for sort-merge joins
(Namit Jain via Ashutosh Chauhan)

[hashutosh] Preparing for 0.12 development

[hashutosh] HIVE-4323 : sqlline dependency is not required (Thejas Nair via 
Ashutosh Chauhan)

[namit] HIVE-4336 Selecting from a view, and another view that also selects 
from that view fails
(Kevin Wilfong via namit)


Changes for Build #2058
[namit] HIVE-4241 optimize hive.enforce.sorting and hive.enforce bucketing join
(Namit Jain via Gang Tim Liu)


Changes for Build #2059
[namit] HIVE-4294 Single sourced multi query cannot handle lateral view
(Navis via namit)


Changes for Build #2060

Changes for Build #2061
[hashutosh] HIVE-4339 : build fails after branch (hcatalog version not updated) 
(Gunther Hagleitner via Ashutosh Chauhan)


Changes for Build #2062
[hashutosh] HIVE-4296 : ant thriftif  fails on  hcatalog (Roshan Naik via 
Ashutosh Chauhan)

[gangtimliu] HIVE-4322: SkewedInfo in Metastore Thrift API cannot be 
deserialized in Python (Samuel Yuan via Gang Tim Liu)


Changes for Build #2063

Changes for Build #2064

Changes for Build #2065
[navis] HIVE-3179 HBase Handler doesn't handle NULLs properly (Lars Francke via 
Navis)


Changes for Build #2066
[cws] HIVE-4268. Beeline should support the -f option (Rob Weltman via cws)


Changes for Build #2067
[hashutosh] HIVE-4275 : Hive does not differentiate scheme and authority in 
file uris (Vikram Dixit via Ashutosh Chauhan)

[hashutosh] HIVE-4352 : Guava not getting included in build package (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4315 : enable doAs in unsecure mode for hive server2, when MR 
job runs locally (Thejas Nair via Ashutosh Chauhan)

[hashutosh] HIVE-4105 : Hive MapJoinOperator unnecessarily deserializes values 
for all join-keys (Vinod KV via Ashutosh Chauhan)

[hashutosh] HIVE-4327 : N

[jira] [Resolved] (HIVE-4019) Ability to create and drop temporary partition function

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan resolved HIVE-4019.


   Resolution: Fixed
Fix Version/s: 0.12.0

Committed to trunk. Thanks, Brock!

> Ability to create and drop temporary partition function
> ---
>
> Key: HIVE-4019
> URL: https://issues.apache.org/jira/browse/HIVE-4019
> Project: Hive
>  Issue Type: New Feature
>  Components: PTF-Windowing
>Reporter: Ashutosh Chauhan
>Assignee: Brock Noland
> Fix For: 0.12.0
>
> Attachments: HIVE-4019-1.patch, HIVE-4019.2.patch, HIVE-4019-3.patch, 
> HIVE-4019-4.patch, HIVE-4019-5.patch, HIVE-4019-6.patch, HIVE-4019-7.patch, 
> hive-4019.q
>
>
> Just like udf/udaf/udtf functions, user should be able to add and drop custom 
> partitioning functions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4398) HS2 Resource leak: operation handles not cleaned when originating session is closed

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-4398:
---

   Resolution: Fixed
Fix Version/s: (was: 0.11.0)
   0.12.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Ashish!

> HS2 Resource leak: operation handles not cleaned when originating session is 
> closed
> ---
>
> Key: HIVE-4398
> URL: https://issues.apache.org/jira/browse/HIVE-4398
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.10.0
>Reporter: Ashish Vaidya
>Assignee: Ashish Vaidya
> Fix For: 0.12.0
>
> Attachments: HIVE-4398.branch-0.11.patch, 
> HIVE-4398.branch-0.11.patch.1, HIVE-4398.branch-0.11.patch.2, 
> HIVE-4398.branch-0.11.patch.3
>
>
> In HS2 closing of sessions doesn't lead to closing of all the operation 
> handles that the session had opened. This JIRA is meant to address this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4423) Improve RCFile::sync(long) 10x

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-4423:
---

   Resolution: Fixed
Fix Version/s: (was: 0.11.0)
   0.12.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Gopal!

> Improve RCFile::sync(long) 10x
> --
>
> Key: HIVE-4423
> URL: https://issues.apache.org/jira/browse/HIVE-4423
> Project: Hive
>  Issue Type: Improvement
> Environment: Ubuntu LXC (1 SSD, 1 disk, 32 gigs of RAM)
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
>  Labels: optimization
> Fix For: 0.12.0
>
> Attachments: HIVE-4423.patch
>
>
> RCFile::sync(long) takes approx ~1 second everytime it gets called because of 
> the inner loops in the function.
> From what was observed with HDFS-4710, single byte reads are an order of 
> magnitude slower than larger 512 byte buffer reads. 
> Even when disk I/O is buffered to this size, there is overhead due to the 
> synchronized read() methods in BlockReaderLocal & RemoteBlockReader classes.
> Removing the readByte() calls in RCFile.sync(long) with a readFully(512 byte) 
> call will speed this function >10x.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4435) Column stats: Distinct value estimator should use hash functions that are pairwise independent

[Shreepadma Venugopalan (JIRA)]

Shreepadma Venugopalan created HIVE-4435:


 Summary: Column stats: Distinct value estimator should use hash 
functions that are pairwise independent
 Key: HIVE-4435
 URL: https://issues.apache.org/jira/browse/HIVE-4435
 Project: Hive
  Issue Type: Bug
  Components: Statistics
Affects Versions: 0.10.0
Reporter: Shreepadma Venugopalan
Assignee: Shreepadma Venugopalan




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HIVE-4428) Misspelling in describe extended output

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates resolved HIVE-4428.
--

Resolution: Won't Fix

What looked like a trivial misspelling turned out to be a compatibility issue.  
Since this is the actual key used in the metastore to record the collection 
delimiter changing the value would affect all existing tables.  Given this it 
seems better to allow the misspelling to remain.

> Misspelling in describe extended output
> ---
>
> Key: HIVE-4428
> URL: https://issues.apache.org/jira/browse/HIVE-4428
> Project: Hive
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 0.11.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Trivial
> Fix For: 0.12.0
>
>
> The output of "describe extended" includes: colelction.delim.  This should be 
> collection.delim.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643768#comment-13643768
 ] 

Gopal V commented on HIVE-4068:
---

Missed svn add of ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java 
before check-in?





> Size of aggregation buffer which uses non-primitive type is not estimated 
> correctly
> ---
>
> Key: HIVE-4068
> URL: https://issues.apache.org/jira/browse/HIVE-4068
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.12.0
>
> Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, 
> HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch
>
>
> Currently, hive assumes an aggregation buffer which holds a map is occupying 
> just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can 
> be thrown (especially for >1k buffer). 
> workaround : set hive.map.aggr.hash.percentmemory= default(0.5)>

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4436) hive.exec.parallel=true doesn't work on hadoop-2

[Gopal V (JIRA)]

Gopal V created HIVE-4436:
-

 Summary: hive.exec.parallel=true doesn't work on hadoop-2
 Key: HIVE-4436
 URL: https://issues.apache.org/jira/browse/HIVE-4436
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
 Environment: Ubuntu LXC (hive-trunk)
Reporter: Gopal V


While running a hive query with multiple independent stages, hive.exec.parallel 
is a valid optimization to use.

The query tested has 3 MR jobs - the first job is the root dependency and the 2 
further job depend on the first one.

When hive.exec.parallel is turned on, the job fails with the following exception

{code}
java.io.IOException: java.lang.InterruptedException
at org.apache.hadoop.ipc.Client.call(Client.java:1214)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy12.mkdirs(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy12.mkdirs(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:447)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2165)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2136)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:544)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1916)
at 
org.apache.hadoop.hive.ql.exec.ExecDriver.createTmpDirs(ExecDriver.java:222)
at 
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:444)
at 
org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
Caused by: java.lang.InterruptedException
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:921)
at org.apache.hadoop.ipc.Client.call(Client.java:1208)
{code}

The query plan is as follows

{code}
  Stage-9 is a root stage
  Stage-8 depends on stages: Stage-9
  Stage-3 depends on stages: Stage-8
  Stage-0 depends on stages: Stage-3
  Stage-4 depends on stages: Stage-0
  Stage-5 depends on stages: Stage-8
  Stage-1 depends on stages: Stage-5
  Stage-6 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-9
Map Reduce Local Work
  Stage: Stage-8
Map Reduce
Map Join Operator
  Stage: Stage-3
Map Reduce
  Stage: Stage-0
Move Operator
  Stage: Stage-4
Stats-Aggr Operator
  Stage: Stage-5
Map Reduce
  Stage: Stage-1
Move Operator
  Stage: Stage-6
Stats-Aggr Operator
{code}

I cannot conclude that this is purely a hive issue, will file a bug on HDFS if 
that does show up during triage.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HIVE-4436) hive.exec.parallel=true doesn't work on hadoop-2

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V reassigned HIVE-4436:
-

Assignee: Gopal V

> hive.exec.parallel=true doesn't work on hadoop-2
> 
>
> Key: HIVE-4436
> URL: https://issues.apache.org/jira/browse/HIVE-4436
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
> Environment: Ubuntu LXC (hive-trunk)
>Reporter: Gopal V
>Assignee: Gopal V
>
> While running a hive query with multiple independent stages, 
> hive.exec.parallel is a valid optimization to use.
> The query tested has 3 MR jobs - the first job is the root dependency and the 
> 2 further job depend on the first one.
> When hive.exec.parallel is turned on, the job fails with the following 
> exception
> {code}
> java.io.IOException: java.lang.InterruptedException
>   at org.apache.hadoop.ipc.Client.call(Client.java:1214)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>   at $Proxy12.mkdirs(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
>   at $Proxy12.mkdirs(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:447)
>   at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2165)
>   at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2136)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:544)
>   at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1916)
>   at 
> org.apache.hadoop.hive.ql.exec.ExecDriver.createTmpDirs(ExecDriver.java:222)
>   at 
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:444)
>   at 
> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>   at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
> Caused by: java.lang.InterruptedException
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279)
>   at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>   at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:921)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1208)
> {code}
> The query plan is as follows
> {code}
>   Stage-9 is a root stage
>   Stage-8 depends on stages: Stage-9
>   Stage-3 depends on stages: Stage-8
>   Stage-0 depends on stages: Stage-3
>   Stage-4 depends on stages: Stage-0
>   Stage-5 depends on stages: Stage-8
>   Stage-1 depends on stages: Stage-5
>   Stage-6 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-9
> Map Reduce Local Work
>   Stage: Stage-8
> Map Reduce
> Map Join Operator
>   Stage: Stage-3
> Map Reduce
>   Stage: Stage-0
> Move Operator
>   Stage: Stage-4
> Stats-Aggr Operator
>   Stage: Stage-5
> Map Reduce
>   Stage: Stage-1
> Move Operator
>   Stage: Stage-6
> Stats-Aggr Operator
> {code}
> I cannot conclude that this is purely a hive issue, will file a bug on HDFS 
> if that does show up during triage.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4436) hive.exec.parallel=true doesn't work on hadoop-2

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-4436:
--

Attachment: HIVE-4436-test.tgz

Small self-contained test.

To use, run 

hive -i create.sql -f test.sql

> hive.exec.parallel=true doesn't work on hadoop-2
> 
>
> Key: HIVE-4436
> URL: https://issues.apache.org/jira/browse/HIVE-4436
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
> Environment: Ubuntu LXC (hive-trunk)
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-4436-test.tgz
>
>
> While running a hive query with multiple independent stages, 
> hive.exec.parallel is a valid optimization to use.
> The query tested has 3 MR jobs - the first job is the root dependency and the 
> 2 further job depend on the first one.
> When hive.exec.parallel is turned on, the job fails with the following 
> exception
> {code}
> java.io.IOException: java.lang.InterruptedException
>   at org.apache.hadoop.ipc.Client.call(Client.java:1214)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>   at $Proxy12.mkdirs(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
>   at $Proxy12.mkdirs(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:447)
>   at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2165)
>   at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2136)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:544)
>   at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1916)
>   at 
> org.apache.hadoop.hive.ql.exec.ExecDriver.createTmpDirs(ExecDriver.java:222)
>   at 
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:444)
>   at 
> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>   at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
> Caused by: java.lang.InterruptedException
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279)
>   at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>   at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:921)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1208)
> {code}
> The query plan is as follows
> {code}
>   Stage-9 is a root stage
>   Stage-8 depends on stages: Stage-9
>   Stage-3 depends on stages: Stage-8
>   Stage-0 depends on stages: Stage-3
>   Stage-4 depends on stages: Stage-0
>   Stage-5 depends on stages: Stage-8
>   Stage-1 depends on stages: Stage-5
>   Stage-6 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-9
> Map Reduce Local Work
>   Stage: Stage-8
> Map Reduce
> Map Join Operator
>   Stage: Stage-3
> Map Reduce
>   Stage: Stage-0
> Move Operator
>   Stage: Stage-4
> Stats-Aggr Operator
>   Stage: Stage-5
> Map Reduce
>   Stage: Stage-1
> Move Operator
>   Stage: Stage-6
> Stats-Aggr Operator
> {code}
> I cannot conclude that this is purely a hive issue, will file a bug on HDFS 
> if that does show up during triage.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4019) Ability to create and drop temporary partition function

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643773#comment-13643773
 ] 

Hudson commented on HIVE-4019:
--

Integrated in Hive-trunk-h0.21 #2082 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/2082/])
HIVE-4019 : Ability to create and drop temporary partition function (Brock 
Noland via Ashutosh Chauhan) (Revision 1476646)

 Result = FAILURE
hashutosh : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476646
Files : 
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/PTFFunctionInfo.java
* /hive/trunk/ql/src/test/queries/clientpositive/ptf_register_tblfn.q
* /hive/trunk/ql/src/test/results/clientpositive/ptf_register_tblfn.q.out
* /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out


> Ability to create and drop temporary partition function
> ---
>
> Key: HIVE-4019
> URL: https://issues.apache.org/jira/browse/HIVE-4019
> Project: Hive
>  Issue Type: New Feature
>  Components: PTF-Windowing
>Reporter: Ashutosh Chauhan
>Assignee: Brock Noland
> Fix For: 0.12.0
>
> Attachments: HIVE-4019-1.patch, HIVE-4019.2.patch, HIVE-4019-3.patch, 
> HIVE-4019-4.patch, HIVE-4019-5.patch, HIVE-4019-6.patch, HIVE-4019-7.patch, 
> hive-4019.q
>
>
> Just like udf/udaf/udtf functions, user should be able to add and drop custom 
> partitioning functions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4423) Improve RCFile::sync(long) 10x

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643774#comment-13643774
 ] 

Hudson commented on HIVE-4423:
--

Integrated in Hive-trunk-h0.21 #2082 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/2082/])
HIVE-4423 : Improve RCFile::sync(long) 10x (Gopal V via Ashutosh Chauhan) 
(Revision 1476648)

 Result = FAILURE
hashutosh : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476648
Files : 
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java


> Improve RCFile::sync(long) 10x
> --
>
> Key: HIVE-4423
> URL: https://issues.apache.org/jira/browse/HIVE-4423
> Project: Hive
>  Issue Type: Improvement
> Environment: Ubuntu LXC (1 SSD, 1 disk, 32 gigs of RAM)
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
>  Labels: optimization
> Fix For: 0.12.0
>
> Attachments: HIVE-4423.patch
>
>
> RCFile::sync(long) takes approx ~1 second everytime it gets called because of 
> the inner loops in the function.
> From what was observed with HDFS-4710, single byte reads are an order of 
> magnitude slower than larger 512 byte buffer reads. 
> Even when disk I/O is buffered to this size, there is overhead due to the 
> synchronized read() methods in BlockReaderLocal & RemoteBlockReader classes.
> Removing the readByte() calls in RCFile.sync(long) with a readFully(512 byte) 
> call will speed this function >10x.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4398) HS2 Resource leak: operation handles not cleaned when originating session is closed

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643775#comment-13643775
 ] 

Hudson commented on HIVE-4398:
--

Integrated in Hive-trunk-h0.21 #2082 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/2082/])
HIVE-4398 : HS2 Resource leak: operation handles not cleaned when 
originating session is closed (Ashish Vaidya via Ashutosh Chauhan) (Revision 
1476647)

 Result = FAILURE
hashutosh : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476647
Files : 
* 
/hive/trunk/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java


> HS2 Resource leak: operation handles not cleaned when originating session is 
> closed
> ---
>
> Key: HIVE-4398
> URL: https://issues.apache.org/jira/browse/HIVE-4398
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.10.0
>Reporter: Ashish Vaidya
>Assignee: Ashish Vaidya
> Fix For: 0.12.0
>
> Attachments: HIVE-4398.branch-0.11.patch, 
> HIVE-4398.branch-0.11.patch.1, HIVE-4398.branch-0.11.patch.2, 
> HIVE-4398.branch-0.11.patch.3
>
>
> In HS2 closing of sessions doesn't lead to closing of all the operation 
> handles that the session had opened. This JIRA is meant to address this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-h0.21 - Build # 2082 - Still Failing

[Apache Jenkins Server]

Changes for Build #2049
[hashutosh] HIVE-3985 : Update new UDAFs introduced for Windowing to work with 
new Decimal Type (Brock Noland via Ashutosh Chauhan)

[hashutosh] HIVE-3840 : hive cli null representation in output is inconsistent 
(Thejas Nair via Ashutosh Chauhan)

[hashutosh] HIVE-4262 : fix last_value UDAF behavior (Harish Butani via 
Ashutosh Chauhan)

[hashutosh] HIVE-4292 : hiveserver2 should support -hiveconf commandline 
parameter (Thejas Nair via Ashutosh Chauhan)


Changes for Build #2050
[hashutosh] HIVE-3908 : create view statement's outputs contains the view and a 
temporary dir. (Prasad Mujumdar via Ashutosh Chauhan)

[hashutosh] HIVE-4302 : Fix how RowSchema and RowResolver are set on 
ReduceSinkOp that precedes PTFOp (Harish Butani via Ashutosh Chauhan)


Changes for Build #2051
[hashutosh] HIVE-3992 : Hive RCFile::sync(long) does a sub-sequence linear 
search for sync blocks (Gopal V via Ashutosh Chauhan)


Changes for Build #2052

Changes for Build #2053
[navis] Missing test results from HIVE-1953 (Vikram Dixit K via Navis)

[namit] HIVE-4314 Result of mapjoin_test_outer.q is not deterministic
(Navis via namit)

[navis] HIVE-1953 Hive should process comments in CliDriver (Vikram Dixit K via 
Navis)

[navis] HIVE-3308 Mixing avro and snappy gives null values (Bennie Schut via 
Navis)

[hashutosh] HIVE-4311 : DOS line endings in auto_join26.q (Gunther Hagleitner 
via Ashutosh Chauhan)

[hashutosh] HIVE-2340 : optimize orderby followed by a groupby (Navis via 
Ashutosh Chauhan)


Changes for Build #2054
[khorgath] HCATALOG-632 Fixing ORC File usage with HCatalog


Changes for Build #2055
[hashutosh] HIVE-4107 : Update Hive 0.10.0 RELEASE_NOTES.txt (Thejas Nair via 
Ashutosh Chauhan)

[hashutosh] HIVE-4271 : Limit precision of decimal type (Gunther Hagleitner via 
Ashutosh Chauhan)

[hashutosh] HIVE-4319 : Revert changes checked-in as part of 1953 (Vikram Dixit 
via Ashutosh Chauhan)


Changes for Build #2056
[hashutosh] HIVE-4078 : Delay the serialize-deserialize pair in 
CommonJoinTaskDispatcher (Gopal V via Ashutosh Chauhan)

[gangtimliu] HIVE-4337: Update list bucketing test results (Samuel Yuan via 
Gang Tim Liu)

[hashutosh] HIVE-4306 : PTFDeserializer should reconstruct OIs based on InputOI 
passed to PTFOperator (Harish Butani and Prajakta Kalmegh via Ashutosh Chauhan)

[hashutosh] HIVE-4334 : ctas test on hadoop 2 has outdated golden file (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4332 : orc*.q tests fail on hadoop 2 (Gunther Hagleitner via 
Ashutosh Chauhan)

[hashutosh] HIVE-4120 : Implement decimal encoding for ORC (Owen Omalley via 
Ashutosh Chauhan)

[hashutosh] HIVE-4303 : 2 empty java files in hcatalog (Owen Omalley via 
Ashutosh Chauhan)

[hashutosh] HIVE-4335 : serde_regex test fails on hadoop 2 (Gunther Hagleitner 
via Ashutosh Chauhan)

[hashutosh] HIVE-4325 : Merge HCat NOTICE file with Hive NOTICE file (Alan 
Gates via Ashutosh Chauhan)

[gates] Moving CHANGES.txt from under trunk to hcatalog-historical

[gates] HIVE-4236 - Clean up remaining items in hive/hcatalog/historical/trunk 
(gates)


Changes for Build #2057
[namit] HIVE-4328 Change default bigtable selection policy for sort-merge joins
(Namit Jain via Ashutosh Chauhan)

[hashutosh] Preparing for 0.12 development

[hashutosh] HIVE-4323 : sqlline dependency is not required (Thejas Nair via 
Ashutosh Chauhan)

[namit] HIVE-4336 Selecting from a view, and another view that also selects 
from that view fails
(Kevin Wilfong via namit)


Changes for Build #2058
[namit] HIVE-4241 optimize hive.enforce.sorting and hive.enforce bucketing join
(Namit Jain via Gang Tim Liu)


Changes for Build #2059
[namit] HIVE-4294 Single sourced multi query cannot handle lateral view
(Navis via namit)


Changes for Build #2060

Changes for Build #2061
[hashutosh] HIVE-4339 : build fails after branch (hcatalog version not updated) 
(Gunther Hagleitner via Ashutosh Chauhan)


Changes for Build #2062
[hashutosh] HIVE-4296 : ant thriftif  fails on  hcatalog (Roshan Naik via 
Ashutosh Chauhan)

[gangtimliu] HIVE-4322: SkewedInfo in Metastore Thrift API cannot be 
deserialized in Python (Samuel Yuan via Gang Tim Liu)


Changes for Build #2063

Changes for Build #2064

Changes for Build #2065
[navis] HIVE-3179 HBase Handler doesn't handle NULLs properly (Lars Francke via 
Navis)


Changes for Build #2066
[cws] HIVE-4268. Beeline should support the -f option (Rob Weltman via cws)


Changes for Build #2067
[hashutosh] HIVE-4275 : Hive does not differentiate scheme and authority in 
file uris (Vikram Dixit via Ashutosh Chauhan)

[hashutosh] HIVE-4352 : Guava not getting included in build package (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4315 : enable doAs in unsecure mode for hive server2, when MR 
job runs locally (Thejas Nair via Ashutosh Chauhan)

[hashutosh] HIVE-4105 : Hive MapJoinOperator unnecessarily deserializes values 
for all join-keys (Vinod KV via Ashutosh Chauhan)

[hashutosh] HIVE-4327 : N

[jira] [Commented] (HIVE-4019) Ability to create and drop temporary partition function

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643777#comment-13643777
 ] 

Hudson commented on HIVE-4019:
--

Integrated in Hive-trunk-hadoop2 #179 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop2/179/])
HIVE-4019 : Ability to create and drop temporary partition function (Brock 
Noland via Ashutosh Chauhan) (Revision 1476646)

 Result = FAILURE
hashutosh : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476646
Files : 
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/PTFFunctionInfo.java
* /hive/trunk/ql/src/test/queries/clientpositive/ptf_register_tblfn.q
* /hive/trunk/ql/src/test/results/clientpositive/ptf_register_tblfn.q.out
* /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out


> Ability to create and drop temporary partition function
> ---
>
> Key: HIVE-4019
> URL: https://issues.apache.org/jira/browse/HIVE-4019
> Project: Hive
>  Issue Type: New Feature
>  Components: PTF-Windowing
>Reporter: Ashutosh Chauhan
>Assignee: Brock Noland
> Fix For: 0.12.0
>
> Attachments: HIVE-4019-1.patch, HIVE-4019.2.patch, HIVE-4019-3.patch, 
> HIVE-4019-4.patch, HIVE-4019-5.patch, HIVE-4019-6.patch, HIVE-4019-7.patch, 
> hive-4019.q
>
>
> Just like udf/udaf/udtf functions, user should be able to add and drop custom 
> partitioning functions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4423) Improve RCFile::sync(long) 10x

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643778#comment-13643778
 ] 

Hudson commented on HIVE-4423:
--

Integrated in Hive-trunk-hadoop2 #179 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop2/179/])
HIVE-4423 : Improve RCFile::sync(long) 10x (Gopal V via Ashutosh Chauhan) 
(Revision 1476648)

 Result = FAILURE
hashutosh : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476648
Files : 
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java


> Improve RCFile::sync(long) 10x
> --
>
> Key: HIVE-4423
> URL: https://issues.apache.org/jira/browse/HIVE-4423
> Project: Hive
>  Issue Type: Improvement
> Environment: Ubuntu LXC (1 SSD, 1 disk, 32 gigs of RAM)
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
>  Labels: optimization
> Fix For: 0.12.0
>
> Attachments: HIVE-4423.patch
>
>
> RCFile::sync(long) takes approx ~1 second everytime it gets called because of 
> the inner loops in the function.
> From what was observed with HDFS-4710, single byte reads are an order of 
> magnitude slower than larger 512 byte buffer reads. 
> Even when disk I/O is buffered to this size, there is overhead due to the 
> synchronized read() methods in BlockReaderLocal & RemoteBlockReader classes.
> Removing the readByte() calls in RCFile.sync(long) with a readFully(512 byte) 
> call will speed this function >10x.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4398) HS2 Resource leak: operation handles not cleaned when originating session is closed

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643779#comment-13643779
 ] 

Hudson commented on HIVE-4398:
--

Integrated in Hive-trunk-hadoop2 #179 (See 
[https://builds.apache.org/job/Hive-trunk-hadoop2/179/])
HIVE-4398 : HS2 Resource leak: operation handles not cleaned when 
originating session is closed (Ashish Vaidya via Ashutosh Chauhan) (Revision 
1476647)

 Result = FAILURE
hashutosh : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1476647
Files : 
* 
/hive/trunk/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java


> HS2 Resource leak: operation handles not cleaned when originating session is 
> closed
> ---
>
> Key: HIVE-4398
> URL: https://issues.apache.org/jira/browse/HIVE-4398
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.10.0
>Reporter: Ashish Vaidya
>Assignee: Ashish Vaidya
> Fix For: 0.12.0
>
> Attachments: HIVE-4398.branch-0.11.patch, 
> HIVE-4398.branch-0.11.patch.1, HIVE-4398.branch-0.11.patch.2, 
> HIVE-4398.branch-0.11.patch.3
>
>
> In HS2 closing of sessions doesn't lead to closing of all the operation 
> handles that the session had opened. This JIRA is meant to address this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-hadoop2 - Build # 179 - Still Failing

[Apache Jenkins Server]

Changes for Build #143

Changes for Build #144
[hashutosh] HIVE-3908 : create view statement's outputs contains the view and a 
temporary dir. (Prasad Mujumdar via Ashutosh Chauhan)

[hashutosh] HIVE-4302 : Fix how RowSchema and RowResolver are set on 
ReduceSinkOp that precedes PTFOp (Harish Butani via Ashutosh Chauhan)


Changes for Build #145
[hashutosh] HIVE-3992 : Hive RCFile::sync(long) does a sub-sequence linear 
search for sync blocks (Gopal V via Ashutosh Chauhan)


Changes for Build #146

Changes for Build #147
[hashutosh] HIVE-2340 : optimize orderby followed by a groupby (Navis via 
Ashutosh Chauhan)


Changes for Build #148
[navis] Missing test results from HIVE-1953 (Vikram Dixit K via Navis)

[namit] HIVE-4314 Result of mapjoin_test_outer.q is not deterministic
(Navis via namit)

[navis] HIVE-1953 Hive should process comments in CliDriver (Vikram Dixit K via 
Navis)

[navis] HIVE-3308 Mixing avro and snappy gives null values (Bennie Schut via 
Navis)

[hashutosh] HIVE-4311 : DOS line endings in auto_join26.q (Gunther Hagleitner 
via Ashutosh Chauhan)


Changes for Build #149
[hashutosh] HIVE-4107 : Update Hive 0.10.0 RELEASE_NOTES.txt (Thejas Nair via 
Ashutosh Chauhan)

[hashutosh] HIVE-4271 : Limit precision of decimal type (Gunther Hagleitner via 
Ashutosh Chauhan)

[hashutosh] HIVE-4319 : Revert changes checked-in as part of 1953 (Vikram Dixit 
via Ashutosh Chauhan)

[khorgath] HCATALOG-632 Fixing ORC File usage with HCatalog


Changes for Build #150
[hashutosh] HIVE-4325 : Merge HCat NOTICE file with Hive NOTICE file (Alan 
Gates via Ashutosh Chauhan)

[gates] Moving CHANGES.txt from under trunk to hcatalog-historical

[gates] HIVE-4236 - Clean up remaining items in hive/hcatalog/historical/trunk 
(gates)


Changes for Build #151
[hashutosh] HIVE-4078 : Delay the serialize-deserialize pair in 
CommonJoinTaskDispatcher (Gopal V via Ashutosh Chauhan)

[gangtimliu] HIVE-4337: Update list bucketing test results (Samuel Yuan via 
Gang Tim Liu)

[hashutosh] HIVE-4306 : PTFDeserializer should reconstruct OIs based on InputOI 
passed to PTFOperator (Harish Butani and Prajakta Kalmegh via Ashutosh Chauhan)

[hashutosh] HIVE-4334 : ctas test on hadoop 2 has outdated golden file (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4332 : orc*.q tests fail on hadoop 2 (Gunther Hagleitner via 
Ashutosh Chauhan)

[hashutosh] HIVE-4120 : Implement decimal encoding for ORC (Owen Omalley via 
Ashutosh Chauhan)

[hashutosh] HIVE-4303 : 2 empty java files in hcatalog (Owen Omalley via 
Ashutosh Chauhan)

[hashutosh] HIVE-4335 : serde_regex test fails on hadoop 2 (Gunther Hagleitner 
via Ashutosh Chauhan)


Changes for Build #152
[namit] HIVE-4328 Change default bigtable selection policy for sort-merge joins
(Namit Jain via Ashutosh Chauhan)

[hashutosh] Preparing for 0.12 development

[hashutosh] HIVE-4323 : sqlline dependency is not required (Thejas Nair via 
Ashutosh Chauhan)

[namit] HIVE-4336 Selecting from a view, and another view that also selects 
from that view fails
(Kevin Wilfong via namit)


Changes for Build #153
[namit] HIVE-4241 optimize hive.enforce.sorting and hive.enforce bucketing join
(Namit Jain via Gang Tim Liu)


Changes for Build #154
[namit] HIVE-4294 Single sourced multi query cannot handle lateral view
(Navis via namit)


Changes for Build #155

Changes for Build #156
[hashutosh] HIVE-4339 : build fails after branch (hcatalog version not updated) 
(Gunther Hagleitner via Ashutosh Chauhan)


Changes for Build #157
[hashutosh] HIVE-4296 : ant thriftif  fails on  hcatalog (Roshan Naik via 
Ashutosh Chauhan)

[gangtimliu] HIVE-4322: SkewedInfo in Metastore Thrift API cannot be 
deserialized in Python (Samuel Yuan via Gang Tim Liu)


Changes for Build #158

Changes for Build #159

Changes for Build #160
[navis] HIVE-3179 HBase Handler doesn't handle NULLs properly (Lars Francke via 
Navis)


Changes for Build #161
[cws] HIVE-4268. Beeline should support the -f option (Rob Weltman via cws)


Changes for Build #162
[hashutosh] HIVE-4275 : Hive does not differentiate scheme and authority in 
file uris (Vikram Dixit via Ashutosh Chauhan)

[hashutosh] HIVE-4352 : Guava not getting included in build package (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4315 : enable doAs in unsecure mode for hive server2, when MR 
job runs locally (Thejas Nair via Ashutosh Chauhan)

[hashutosh] HIVE-4105 : Hive MapJoinOperator unnecessarily deserializes values 
for all join-keys (Vinod KV via Ashutosh Chauhan)

[hashutosh] HIVE-4327 : NPE in constant folding with decimal (Gunther 
Hagleitner via Ashutosh Chauhan)

[namit] HIVE-4167 Hive converts bucket map join to SMB join even when tables 
are not sorted
(Namit Jain via Ashutosh)

[namit] HIVE-4308 Newly added test TestCliDriver.hiveprofiler_union0 is failing 
on trunk
(Navis via namit)

[namit] HIVE-3996 Correctly enforce the memory limit on the multi-table map-join
(Vikram Dixit via namit)

[navis] HIVE-4261 union_remove_10 is fail

Re: Review Request: HIVE-4373 - Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10812/
---

(Updated April 28, 2013, 12:22 a.m.)


Review request for hive.


Changes
---

Reverting the package-info.java location change. Old location works.


Description
---

HIVE-4373


This addresses bug HIVE-4373.
https://issues.apache.org/jira/browse/HIVE-4373


Diffs (updated)
-

  common/build.xml 731f26e 
  jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java 3d1a969 
  jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java 938dd9a 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
18594cb 

Diff: https://reviews.apache.org/r/10812/diff/


Testing
---

Unit test included. Also tested with remote HS2 server.


Thanks,

Thejas Nair

Re: Review Request: HIVE-4373 - Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect

[Thejas Nair]

> On April 27, 2013, 6:59 a.m., Carl Steinbach wrote:
> > common/src/scripts/saveVersion.sh, line 66
> > 
> >
> > I think the output directory was correct to begin with. Please revert.
> 
> Thejas Nair wrote:
> HiveVersionInfo is in org/apache/hive/common/util/, so this 
> package-info.java with the annotations expected by HiveVersionInfo needs to 
> be in same package. Otherwise  HiveVersionInfo will not work.
> Hence the change to move it to this dir. Please let me know if you 
> disagree. 
>

Reverting the package-info.java location change. Old location works.


- Thejas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10812/#review19822
---


On April 28, 2013, 12:22 a.m., Thejas Nair wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10812/
> ---
> 
> (Updated April 28, 2013, 12:22 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Description
> ---
> 
> HIVE-4373
> 
> 
> This addresses bug HIVE-4373.
> https://issues.apache.org/jira/browse/HIVE-4373
> 
> 
> Diffs
> -
> 
>   common/build.xml 731f26e 
>   jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java 3d1a969 
>   jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java 938dd9a 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 18594cb 
> 
> Diff: https://reviews.apache.org/r/10812/diff/
> 
> 
> Testing
> ---
> 
> Unit test included. Also tested with remote HS2 server.
> 
> 
> Thanks,
> 
> Thejas Nair
> 
>

[jira] [Updated] (HIVE-4373) Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-4373:


Attachment: HIVE-4373.3.patch

HIVE-4373.3.patch - patch without the package-info.java location change. Old 
location works as Carl mentioned. This addresses all comments from 
[~cwsteinbach].


> Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is 
> incorrect
> 
>
> Key: HIVE-4373
> URL: https://issues.apache.org/jira/browse/HIVE-4373
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.11.0
>Reporter: Deepesh Khandelwal
>Assignee: Thejas M Nair
>Priority: Minor
> Fix For: 0.11.0
>
> Attachments: HIVE-4373.1.patch, HIVE-4373.2.patch, HIVE-4373.3.patch
>
>
> When running beeline
> {code}
> % beeline -u 'jdbc:hive2://localhost:1' -n hive -p passwd -d 
> org.apache.hive.jdbc.HiveDriver
> Connecting to jdbc:hive2://localhost:1
> Connected to: Hive (version 0.10.0)
> Driver: Hive (version 0.11.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> {code}
> The Hive version in the "Connected to: " string says 0.10.0 instead of 0.11.0.
> Looking at the code it seems that the version is hardcoded at two places:
> line 250 in jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java
> line 833 in jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4373) Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-4373:


Status: Open  (was: Patch Available)

> Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is 
> incorrect
> 
>
> Key: HIVE-4373
> URL: https://issues.apache.org/jira/browse/HIVE-4373
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.11.0
>Reporter: Deepesh Khandelwal
>Assignee: Thejas M Nair
>Priority: Minor
> Fix For: 0.11.0
>
> Attachments: HIVE-4373.1.patch, HIVE-4373.2.patch, HIVE-4373.3.patch
>
>
> When running beeline
> {code}
> % beeline -u 'jdbc:hive2://localhost:1' -n hive -p passwd -d 
> org.apache.hive.jdbc.HiveDriver
> Connecting to jdbc:hive2://localhost:1
> Connected to: Hive (version 0.10.0)
> Driver: Hive (version 0.11.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> {code}
> The Hive version in the "Connected to: " string says 0.10.0 instead of 0.11.0.
> Looking at the code it seems that the version is hardcoded at two places:
> line 250 in jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java
> line 833 in jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4373) Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is incorrect

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-4373:


Status: Patch Available  (was: Open)

> Hive Version returned by HiveDatabaseMetaData.getDatabaseProductVersion is 
> incorrect
> 
>
> Key: HIVE-4373
> URL: https://issues.apache.org/jira/browse/HIVE-4373
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.11.0
>Reporter: Deepesh Khandelwal
>Assignee: Thejas M Nair
>Priority: Minor
> Fix For: 0.11.0
>
> Attachments: HIVE-4373.1.patch, HIVE-4373.2.patch, HIVE-4373.3.patch
>
>
> When running beeline
> {code}
> % beeline -u 'jdbc:hive2://localhost:1' -n hive -p passwd -d 
> org.apache.hive.jdbc.HiveDriver
> Connecting to jdbc:hive2://localhost:1
> Connected to: Hive (version 0.10.0)
> Driver: Hive (version 0.11.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> {code}
> The Hive version in the "Connected to: " string says 0.10.0 instead of 0.11.0.
> Looking at the code it seems that the version is hardcoded at two places:
> line 250 in jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java
> line 833 in jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4437) Missing file on HIVE-4068

[Navis (JIRA)]

Navis created HIVE-4437:
---

 Summary: Missing file on HIVE-4068
 Key: HIVE-4437
 URL: https://issues.apache.org/jira/browse/HIVE-4437
 Project: Hive
  Issue Type: Task
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Critical




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643849#comment-13643849
 ] 

Navis commented on HIVE-4068:
-

[~gopalv] Thanks. I thinks I should not commit things.

> Size of aggregation buffer which uses non-primitive type is not estimated 
> correctly
> ---
>
> Key: HIVE-4068
> URL: https://issues.apache.org/jira/browse/HIVE-4068
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.12.0
>
> Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, 
> HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch
>
>
> Currently, hive assumes an aggregation buffer which holds a map is occupying 
> just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can 
> be thrown (especially for >1k buffer). 
> workaround : set hive.map.aggr.hash.percentmemory= default(0.5)>

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HIVE-4437) Missing file on HIVE-4068

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis resolved HIVE-4437.
-

Resolution: Fixed

Committed to trunk

> Missing file on HIVE-4068
> -
>
> Key: HIVE-4437
> URL: https://issues.apache.org/jira/browse/HIVE-4437
> Project: Hive
>  Issue Type: Task
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Critical
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4436) hive.exec.parallel=true doesn't work on hadoop-2

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643853#comment-13643853
 ] 

Navis commented on HIVE-4436:
-

I saw similar exceptions on long running hive-server sometimes 
(InterruptedException on mkdirs or delete). I couldn't find why those threads 
are interrupted, so made a proxy FileSystem on HDFS, de-interrupting thread 
before calling FS APIs.

> hive.exec.parallel=true doesn't work on hadoop-2
> 
>
> Key: HIVE-4436
> URL: https://issues.apache.org/jira/browse/HIVE-4436
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
> Environment: Ubuntu LXC (hive-trunk)
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-4436-test.tgz
>
>
> While running a hive query with multiple independent stages, 
> hive.exec.parallel is a valid optimization to use.
> The query tested has 3 MR jobs - the first job is the root dependency and the 
> 2 further job depend on the first one.
> When hive.exec.parallel is turned on, the job fails with the following 
> exception
> {code}
> java.io.IOException: java.lang.InterruptedException
>   at org.apache.hadoop.ipc.Client.call(Client.java:1214)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>   at $Proxy12.mkdirs(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
>   at $Proxy12.mkdirs(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:447)
>   at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2165)
>   at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2136)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:544)
>   at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1916)
>   at 
> org.apache.hadoop.hive.ql.exec.ExecDriver.createTmpDirs(ExecDriver.java:222)
>   at 
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:444)
>   at 
> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>   at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
> Caused by: java.lang.InterruptedException
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279)
>   at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>   at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:921)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1208)
> {code}
> The query plan is as follows
> {code}
>   Stage-9 is a root stage
>   Stage-8 depends on stages: Stage-9
>   Stage-3 depends on stages: Stage-8
>   Stage-0 depends on stages: Stage-3
>   Stage-4 depends on stages: Stage-0
>   Stage-5 depends on stages: Stage-8
>   Stage-1 depends on stages: Stage-5
>   Stage-6 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-9
> Map Reduce Local Work
>   Stage: Stage-8
> Map Reduce
> Map Join Operator
>   Stage: Stage-3
> Map Reduce
>   Stage: Stage-0
> Move Operator
>   Stage: Stage-4
> Stats-Aggr Operator
>   Stage: Stage-5
> Map Reduce
>   Stage: Stage-1
> Move Operator
>   Stage: Stage-6
> Stats-Aggr Operator
> {code}
> I cannot conclude that this is purely a hive issue, will file a bug on HDFS 
> if that does show up during triage.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3952) merge map-job followed by map-reduce job

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643867#comment-13643867
 ] 

Ashutosh Chauhan commented on HIVE-3952:


[~vinodkv] It will be great to see this in 0.11. Patch doesn't apply cleanly on 
0.11. Can you provide a backport for 0.11 ?

> merge map-job followed by map-reduce job
> 
>
> Key: HIVE-3952
> URL: https://issues.apache.org/jira/browse/HIVE-3952
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Namit Jain
>Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.12.0
>
> Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, 
> HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, 
> HIVE-3952-20130424.txt
>
>
> Consider the query like:
> select count(*) FROM
> ( select idOne, idTwo, value FROM
>   bigTable   
>   JOIN
> 
>   smallTableOne on (bigTable.idOne = smallTableOne.idOne) 
>   
>   ) firstjoin 
> 
> JOIN  
> 
> smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);
> where smallTableOne and smallTableTwo are smaller than 
> hive.auto.convert.join.noconditionaltask.size and
> hive.auto.convert.join.noconditionaltask is set to true.
> The joins are collapsed into mapjoins, and it leads to a map-only job
> (for the map-joins) followed by a map-reduce job (for the group by).
> Ideally, the map-only job should be merged with the following map-reduce job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4433) Fix C++ Thrift bindings broken in HIVE-4322

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643876#comment-13643876
 ] 

Ashutosh Chauhan commented on HIVE-4433:


[~cwsteinbach] The original culprit HIVE-4322 is not on 0.11. So, I think this 
cannot be marked as blocker for 0.11. Do you agree?

> Fix C++ Thrift bindings broken in HIVE-4322
> ---
>
> Key: HIVE-4433
> URL: https://issues.apache.org/jira/browse/HIVE-4433
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Thrift API
>Affects Versions: 0.11.0
>Reporter: Carl Steinbach
>Assignee: Samuel Yuan
>Priority: Blocker
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643922#comment-13643922
 ] 

Gopal V commented on HIVE-4068:
---

[~navis]: no big deal, man - we all forget stuff :)

> Size of aggregation buffer which uses non-primitive type is not estimated 
> correctly
> ---
>
> Key: HIVE-4068
> URL: https://issues.apache.org/jira/browse/HIVE-4068
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.12.0
>
> Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, 
> HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch
>
>
> Currently, hive assumes an aggregation buffer which holds a map is occupying 
> just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can 
> be thrown (especially for >1k buffer). 
> workaround : set hive.map.aggr.hash.percentmemory= default(0.5)>

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira