[jira] [Commented] (HIVE-3128) use commons-compress instead of forking tar process

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399851#comment-13399851
 ] 

Hudson commented on HIVE-3128:
--

Integrated in Hive-trunk-h0.21 #1506 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/1506/])
HIVE-3128 Use commons-compress instead of forking tar process (Kanna 
Karanam via egc) (Revision 1353044)

 Result = FAILURE
ecapriolo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353044
Files : 
* /hive/trunk/common/ivy.xml
* /hive/trunk/common/src/java/org/apache/hadoop/hive/common/FileUtils.java
* /hive/trunk/ivy/libraries.properties


> use commons-compress instead of forking tar process
> ---
>
> Key: HIVE-3128
> URL: https://issues.apache.org/jira/browse/HIVE-3128
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, Query Processor
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
> Fix For: 0.10.0
>
> Attachments: HIVE-3128.1.patch.txt, HIVE-3128.2.patch.txt
>
>
> TAR tool doesn’t exist by default on windows systems so use the CAB files on 
> windows

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-h0.21 - Build # 1506 - Failure

[Apache Jenkins Server]

Changes for Build #1506
[ecapriolo] HIVE-3128 Use commons-compress instead of forking tar process 
(Kanna Karanam via egc)




1 tests failed.
REGRESSION:  
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1

Error Message:
Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... 
-Dtest.silent=false" to get more logs.

Stack Trace:
junit.framework.AssertionFailedError: Unexpected exception
See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get 
more logs.
at junit.framework.Assert.fail(Assert.java:50)
at 
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:10642)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)




The Apache Jenkins build system has built Hive-trunk-h0.21 (build 
#$BUILD_NUMBER)

Status: Failure

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1506/ to 
view the results.

[jira] [Updated] (HIVE-3178) retry not honored in RetryingRawMetastore

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3178:
-

  Component/s: Metastore
Fix Version/s: 0.10.0

> retry not honored in RetryingRawMetastore
> -
>
> Key: HIVE-3178
> URL: https://issues.apache.org/jira/browse/HIVE-3178
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Namit Jain
>Assignee: Namit Jain
> Fix For: 0.10.0
>
> Attachments: hive.3178.1.patch
>
>
> The retrymetastore catches JDOException, but they are always wrapped by 
> reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3168) LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3168:
-

Component/s: Serializers/Deserializers

> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of 
> underlying BytesWritable
> -
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 0.9.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of 
> the LazyBinary's underlying BytesWritable object, which can be greater than 
> the size of the actual contents. 
> This leads to additional characters at the end of the ByteArrayRef returned. 
> When the LazyBinary object gets re-used, there can be remnants of the later 
> portion of previous entry. 
> This was not seen while reading through hive queries, which I think is 
> because a copy elsewhere seems to create LazyBinary with length == capacity. 
> (probably LazyBinary copy constructor). This was seen when MR or pig used 
> Hcatalog to read the data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HIVE-3168) LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach reopened HIVE-3168:
--


> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of 
> underlying BytesWritable
> -
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 0.9.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of 
> the LazyBinary's underlying BytesWritable object, which can be greater than 
> the size of the actual contents. 
> This leads to additional characters at the end of the ByteArrayRef returned. 
> When the LazyBinary object gets re-used, there can be remnants of the later 
> portion of previous entry. 
> This was not seen while reading through hive queries, which I think is 
> because a copy elsewhere seems to create LazyBinary with length == capacity. 
> (probably LazyBinary copy constructor). This was seen when MR or pig used 
> Hcatalog to read the data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3168) LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3168:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

@Thejas: I left some comments on phabricator. Thanks.

> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of 
> underlying BytesWritable
> -
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of 
> the LazyBinary's underlying BytesWritable object, which can be greater than 
> the size of the actual contents. 
> This leads to additional characters at the end of the ByteArrayRef returned. 
> When the LazyBinary object gets re-used, there can be remnants of the later 
> portion of previous entry. 
> This was not seen while reading through hive queries, which I think is 
> because a copy elsewhere seems to create LazyBinary with length == capacity. 
> (probably LazyBinary copy constructor). This was seen when MR or pig used 
> Hcatalog to read the data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3180) Fix Eclipse classpath template broken in HIVE-3128

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3180:
-

Status: Patch Available  (was: Open)

Review request: https://reviews.apache.org/r/5535/

@Ed: Can you please take a look? Thanks.

> Fix Eclipse classpath template broken in HIVE-3128
> --
>
> Key: HIVE-3180
> URL: https://issues.apache.org/jira/browse/HIVE-3180
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Affects Versions: 0.10.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.10.0
>
> Attachments: HIVE-3180.1.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3180) Fix Eclipse classpath template broken in HIVE-3128

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3180:
-

Attachment: HIVE-3180.1.patch.txt

> Fix Eclipse classpath template broken in HIVE-3128
> --
>
> Key: HIVE-3180
> URL: https://issues.apache.org/jira/browse/HIVE-3180
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Affects Versions: 0.10.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.10.0
>
> Attachments: HIVE-3180.1.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request: HIVE-3180. Fix Eclipse classpath template broken in HIVE-3128

[Carl Steinbach]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5535/
---

Review request for hive and Edward  Capriolo.


Description
---

commit ca753927fa927ffdcf53cd1d4f952fb7a490b27a
Author: Carl Steinbach 
Date:   Fri Jun 22 19:10:25 2012 -0700

HIVE-3180. Fix Eclipse classpath template broken in HIVE-3128

 eclipse-templates/.classpath |1 +
 1 files changed, 1 insertions(+), 0 deletions(-)


This addresses bug HIVE-3180.
https://issues.apache.org/jira/browse/HIVE-3180


Diffs
-

  eclipse-templates/.classpath 0cab940 

Diff: https://reviews.apache.org/r/5535/diff/


Testing
---


Thanks,

Carl Steinbach

[jira] [Created] (HIVE-3180) Fix Eclipse classpath template broken in HIVE-3128

[Carl Steinbach (JIRA)]

Carl Steinbach created HIVE-3180:


 Summary: Fix Eclipse classpath template broken in HIVE-3128
 Key: HIVE-3180
 URL: https://issues.apache.org/jira/browse/HIVE-3180
 Project: Hive
  Issue Type: Bug
  Components: Build Infrastructure
Affects Versions: 0.10.0
Reporter: Carl Steinbach
Assignee: Carl Steinbach
 Fix For: 0.10.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3068) Add ability to export table metadata as JSON on table drop

[Andrew Chalfant (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Chalfant updated HIVE-3068:
--

Status: Patch Available  (was: In Progress)

> Add ability to export table metadata as JSON on table drop
> --
>
> Key: HIVE-3068
> URL: https://issues.apache.org/jira/browse/HIVE-3068
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Serializers/Deserializers
>Reporter: Andrew Chalfant
>Assignee: Andrew Chalfant
>Priority: Minor
>  Labels: features, newbie
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When a table is dropped, the contents go to the users trash but the metadata 
> is lost. It would be super neat to be able to save the metadata as well so 
> that tables could be trivially re-instantiated via thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3068) Add ability to export table metadata as JSON on table drop

[Andrew Chalfant (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399799#comment-13399799
 ] 

Andrew Chalfant commented on HIVE-3068:
---

Diff available here: https://reviews.facebook.net/D3795

> Add ability to export table metadata as JSON on table drop
> --
>
> Key: HIVE-3068
> URL: https://issues.apache.org/jira/browse/HIVE-3068
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Serializers/Deserializers
>Reporter: Andrew Chalfant
>Assignee: Andrew Chalfant
>Priority: Minor
>  Labels: features, newbie
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When a table is dropped, the contents go to the users trash but the metadata 
> is lost. It would be super neat to be able to save the metadata as well so 
> that tables could be trivially re-instantiated via thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Work started] (HIVE-3068) Add ability to export table metadata as JSON on table drop

[Andrew Chalfant (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-3068 started by Andrew Chalfant.

> Add ability to export table metadata as JSON on table drop
> --
>
> Key: HIVE-3068
> URL: https://issues.apache.org/jira/browse/HIVE-3068
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Serializers/Deserializers
>Reporter: Andrew Chalfant
>Assignee: Andrew Chalfant
>Priority: Minor
>  Labels: features, newbie
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When a table is dropped, the contents go to the users trash but the metadata 
> is lost. It would be super neat to be able to save the metadata as well so 
> that tables could be trivially re-instantiated via thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3178) retry not honored in RetryingRawMetastore

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3178:


Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed, thanks Namit.

> retry not honored in RetryingRawMetastore
> -
>
> Key: HIVE-3178
> URL: https://issues.apache.org/jira/browse/HIVE-3178
> Project: Hive
>  Issue Type: Bug
>Reporter: Namit Jain
>Assignee: Namit Jain
> Attachments: hive.3178.1.patch
>
>
> The retrymetastore catches JDOException, but they are always wrapped by 
> reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3092) Hive tests should load Hive classes from build directory, not Ivy cache

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399789#comment-13399789
 ] 

Kevin Wilfong commented on HIVE-3092:
-

No, he never got back to me.

I spoke with Namit Jain who worked with Ning, he didn't think it was needed 
anymore.

If you can remove it and the tests still pass, it's probably ok, otherwise, 
we'll find a deficiency in our tests that should be fixed.

> Hive tests should load Hive classes from build directory, not Ivy cache
> ---
>
> Key: HIVE-3092
> URL: https://issues.apache.org/jira/browse/HIVE-3092
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure, Tests
>Reporter: Jakob Homan
>Assignee: Kevin Wilfong
> Attachments: HIVE-3092.1.patch.txt, HIVE-3092.2.patch.txt
>
>
> As discussed in HIVE-895, currently the tests pull in jars for other 
> components rather from Ivy rather than using the built classes and jars in 
> the build directory (bit.ly/LzndQU).  This means that absent a very-clean, 
> one is testing against a previous version of the code and cross-component 
> tests are invalid.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2021) Add a configuration property that sets the variable substitution max depth

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo updated HIVE-2021:
--

Attachment: hive-2021.patch.txt.1

Rock it.

> Add a configuration property that sets the variable substitution max depth
> --
>
> Key: HIVE-2021
> URL: https://issues.apache.org/jira/browse/HIVE-2021
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration, Query Processor
>Affects Versions: 0.10.0
>Reporter: Carl Steinbach
>Assignee: Edward Capriolo
> Fix For: 0.10.0
>
> Attachments: hive-2021.patch.txt.1
>
>
> The VariableSubstitution class contains a hardcoded MAX_SUBST=40 value which 
> defines the maximum number of variable references that are allowed to appear 
> in a single Hive statement. This value should be configurable via hiveconf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2021) Add a configuration property that sets the variable substitution max depth

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo updated HIVE-2021:
--

Fix Version/s: 0.10.0
Affects Version/s: 0.10.0
   Status: Patch Available  (was: Open)

> Add a configuration property that sets the variable substitution max depth
> --
>
> Key: HIVE-2021
> URL: https://issues.apache.org/jira/browse/HIVE-2021
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration, Query Processor
>Affects Versions: 0.10.0
>Reporter: Carl Steinbach
>Assignee: Edward Capriolo
> Fix For: 0.10.0
>
> Attachments: hive-2021.patch.txt.1
>
>
> The VariableSubstitution class contains a hardcoded MAX_SUBST=40 value which 
> defines the maximum number of variable references that are allowed to appear 
> in a single Hive statement. This value should be configurable via hiveconf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3135) add an option in ptest to run on a single machine

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399778#comment-13399778
 ] 

Hudson commented on HIVE-3135:
--

Integrated in Hive-trunk-h0.21 #1505 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/1505/])
HIVE-3135. add an option in ptest to run on a single machine (Namit Jain 
via kevinwilfong) (Revision 1352973)

 Result = SUCCESS
kevinwilfong : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352973
Files : 
* /hive/trunk/testutils/ptest/hivetest.py


> add an option in ptest to run on a single machine
> -
>
> Key: HIVE-3135
> URL: https://issues.apache.org/jira/browse/HIVE-3135
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Affects Versions: 0.10.0
>Reporter: Namit Jain
>Assignee: Namit Jain
> Fix For: 0.10.0
>
>
> There is no need for any sudo in that case

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399767#comment-13399767
 ] 

Edward Capriolo commented on HIVE-3127:
---

Welcome to hive :) I am willing to bet we only hit the block merge if the input 
is larger then hdfs blocks size, or the defaults is high and it normally does 
not happen during the course of the unit tests.

> Don’t pass–hconf values as command line arguments to child JVM to avoid 
> command line exceeding char limit on windows
> 
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
>  Issue Type: Bug
>  Components: Configuration, Windows
>Affects Versions: 0.9.0, 0.10.0, 0.9.1
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows 
> latest versions http://support.microsoft.com/kb/830473). This limit will be 
> exceeded easily when it appends individual –hconf values to the command 
> string. To work around this problem, Write all changed hconf values to a temp 
> file and pass the temp file path to the child jvm to read and initialize the 
> -hconf parameters from file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3092) Hive tests should load Hive classes from build directory, not Ivy cache

[Kanna Karanam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399766#comment-13399766
 ] 

Kanna Karanam commented on HIVE-3092:
-

Hi Kevin - Any update on this. Thanks

> Hive tests should load Hive classes from build directory, not Ivy cache
> ---
>
> Key: HIVE-3092
> URL: https://issues.apache.org/jira/browse/HIVE-3092
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure, Tests
>Reporter: Jakob Homan
>Assignee: Kevin Wilfong
> Attachments: HIVE-3092.1.patch.txt, HIVE-3092.2.patch.txt
>
>
> As discussed in HIVE-895, currently the tests pull in jars for other 
> components rather from Ivy rather than using the built classes and jars in 
> the build directory (bit.ly/LzndQU).  This means that absent a very-clean, 
> one is testing against a previous version of the code and cross-component 
> tests are invalid.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows

[Kanna Karanam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399754#comment-13399754
 ] 

Kanna Karanam commented on HIVE-3127:
-

Thanks Edward, I will look into it. I wonder how I got 100% test pass. It looks 
like there are no unit tests to test this scenario  or they are passing with 
default conf values. 
Can you please suggest a query to create a unit test for this?
Thanks

> Don’t pass–hconf values as command line arguments to child JVM to avoid 
> command line exceeding char limit on windows
> 
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
>  Issue Type: Bug
>  Components: Configuration, Windows
>Affects Versions: 0.9.0, 0.10.0, 0.9.1
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows 
> latest versions http://support.microsoft.com/kb/830473). This limit will be 
> exceeded easily when it appends individual –hconf values to the command 
> string. To work around this problem, Write all changed hconf values to a temp 
> file and pass the temp file path to the child jvm to read and initialize the 
> -hconf parameters from file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HIVE-2021) Add a configuration property that sets the variable substitution max depth

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo reassigned HIVE-2021:
-

Assignee: Edward Capriolo

> Add a configuration property that sets the variable substitution max depth
> --
>
> Key: HIVE-2021
> URL: https://issues.apache.org/jira/browse/HIVE-2021
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration, Query Processor
>Reporter: Carl Steinbach
>Assignee: Edward Capriolo
>
> The VariableSubstitution class contains a hardcoded MAX_SUBST=40 value which 
> defines the maximum number of variable references that are allowed to appear 
> in a single Hive statement. This value should be configurable via hiveconf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HIVE-3068) Add ability to export table metadata as JSON on table drop

[Andrew Chalfant (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Chalfant reassigned HIVE-3068:
-

Assignee: Andrew Chalfant

> Add ability to export table metadata as JSON on table drop
> --
>
> Key: HIVE-3068
> URL: https://issues.apache.org/jira/browse/HIVE-3068
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Serializers/Deserializers
>Reporter: Andrew Chalfant
>Assignee: Andrew Chalfant
>Priority: Minor
>  Labels: features, newbie
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When a table is dropped, the contents go to the users trash but the metadata 
> is lost. It would be super neat to be able to save the metadata as well so 
> that tables could be trivially re-instantiated via thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3127) Don’t pass–hconf values as command line arguments to child JVM to avoid command line exceeding char limit on windows

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399743#comment-13399743
 ] 

Edward Capriolo commented on HIVE-3127:
---

Kanna,

Your missing something. The BlockMergeTask ask uses -jobconf  . You will need 
to patch that as well.

[edward@tablitha trunk]$ grep -R '\-jobconf' ./*

{noformat}
./ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java:
} else if (args[i].equals("-jobconf")) {
{noformat}

> Don’t pass–hconf values as command line arguments to child JVM to avoid 
> command line exceeding char limit on windows
> 
>
> Key: HIVE-3127
> URL: https://issues.apache.org/jira/browse/HIVE-3127
> Project: Hive
>  Issue Type: Bug
>  Components: Configuration, Windows
>Affects Versions: 0.9.0, 0.10.0, 0.9.1
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3127.1.patch.txt, HIVE-3127.2.patch.txt
>
>
> The maximum length of the DOS command string is 8191 characters (in Windows 
> latest versions http://support.microsoft.com/kb/830473). This limit will be 
> exceeded easily when it appends individual –hconf values to the command 
> string. To work around this problem, Write all changed hconf values to a temp 
> file and pass the temp file path to the child jvm to read and initialize the 
> -hconf parameters from file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3177) ensure the Resultset returned by getTables and getColumns conforms to JDBC 3.0

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3177:
-

Component/s: JDBC

> ensure the Resultset returned by getTables and getColumns conforms to JDBC 3.0
> --
>
> Key: HIVE-3177
> URL: https://issues.apache.org/jira/browse/HIVE-3177
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 0.8.1
>Reporter: N Campbell
>
> The ORACLE JDBC API documentation for 3.0 defines the result set column names 
> and types a driver should be returning for those methods. The Hive driver is 
> returning 5 instead of 10 for getTables and 22 instead of 23 for getColumns. 
> An applicable value should be returned in any column where there is no 
> equivalent concept/metadata object in Hive for one or more of the JDBC 
> documented columns vs not returning it or throwing exceptions etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3126) Generate & build the velocity based Hive tests on windows by fixing the path issues

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399738#comment-13399738
 ] 

Carl Steinbach commented on HIVE-3126:
--

@Kanna: Please create a review request. Thanks.

> Generate & build the velocity based Hive tests on windows by fixing the path 
> issues
> ---
>
> Key: HIVE-3126
> URL: https://issues.apache.org/jira/browse/HIVE-3126
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.9.0, 0.10.0, 0.9.1
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows, test
> Fix For: 0.10.0
>
> Attachments: HIVE-3126.1.patch.txt
>
>
> 1)Escape the backward slash in Canonical Path if unit test runs on windows.
> 2)Diff comparison – 
>  a.   Ignore the extra spacing on windows
>  b.   Ignore the different line endings on windows & Unix
>  c.   Convert the file paths to windows specific. (Handle spaces 
> etc..)
> 3)Set the right file scheme & class path separators while invoking the junit 
> task from 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3176) implement returning values for SQLException getSQLState()

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3176:
-

Component/s: JDBC

> implement returning values for SQLException getSQLState()
> -
>
> Key: HIVE-3176
> URL: https://issues.apache.org/jira/browse/HIVE-3176
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 0.8.1
>Reporter: N Campbell
>
> a dynamic SQL application should be able to check the values returned by 
> getSQLState on a SQLException object. Currently the Hive driver is not doing 
> this (throws exceptions etc).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3146) Support external hive tables whose data are stored in Azure blob store/Azure Storage Volumes (ASV)

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399723#comment-13399723
 ] 

Edward Capriolo commented on HIVE-3146:
---

Kanna, I actually do not believe we need this check at all.

{noformat}
  private String downloadResource(String value, boolean convertToUnix) {
if (value.matches("("+ getMatchingSchemaAsRegex() +")://.*")) {
  try {
FileSystem fs = FileSystem.get(new URI(value), conf);
{noformat}

I just dealt with one of these. 

https://issues.apache.org/jira/browse/HIVE-1444?attachmentSortBy=dateTime

Hadoop and hive are supposed to support pluggable DFS's, we should not have a 
'list of approved fs' anywhere, it just makes more work and more 
incomparability problems.

If I understand this method correctly all we probably need is a check for:

{noformat}
if (!fs.contains("file:///")){ }
{noformat}


Am I right?

> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)
> --
>
> Key: HIVE-3146
> URL: https://issues.apache.org/jira/browse/HIVE-3146
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Affects Versions: 0.10.0
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3146.1.patch.txt
>
>
> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3153) Release codecs and output streams between flushes of RCFile

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-3153:


Attachment: (was: giraph-pom.patch)

> Release codecs and output streams between flushes of RCFile
> ---
>
> Key: HIVE-3153
> URL: https://issues.apache.org/jira/browse/HIVE-3153
> Project: Hive
>  Issue Type: Improvement
>  Components: Compression
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
>
> Currently, RCFile writer holds a compression codec per a file and a 
> compression output stream per a column. Especially for queries that use 
> dynamic partitions this quickly consumes a lot of memory.
> I'd like flushRecords to get a codec from the pool and create the compression 
> output stream in flushRecords.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3153) Release codecs and output streams between flushes of RCFile

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-3153:


Attachment: giraph-pom.patch

> Release codecs and output streams between flushes of RCFile
> ---
>
> Key: HIVE-3153
> URL: https://issues.apache.org/jira/browse/HIVE-3153
> Project: Hive
>  Issue Type: Improvement
>  Components: Compression
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
>
> Currently, RCFile writer holds a compression codec per a file and a 
> compression output stream per a column. Especially for queries that use 
> dynamic partitions this quickly consumes a lot of memory.
> I'd like flushRecords to get a codec from the pool and create the compression 
> output stream in flushRecords.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs

[Andrew Bayer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Bayer updated HIVE-2646:
---

Attachment: BIGTOP-649.diff.txt

> Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
> 
>
> Key: HIVE-2646
> URL: https://issues.apache.org/jira/browse/HIVE-2646
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Affects Versions: 0.8.0
>Reporter: Andrew Bayer
>Assignee: Andrew Bayer
>Priority: Critical
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-2646-fixtests.patch, HIVE-2646-fixtests.txt, 
> HIVE-2646.D2133.1.patch, HIVE-2646.D2133.10.patch, HIVE-2646.D2133.11.patch, 
> HIVE-2646.D2133.12.patch, HIVE-2646.D2133.13.patch, HIVE-2646.D2133.14.patch, 
> HIVE-2646.D2133.15.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, 
> HIVE-2646.D2133.4.patch, HIVE-2646.D2133.5.patch, HIVE-2646.D2133.6.patch, 
> HIVE-2646.D2133.7.patch, HIVE-2646.D2133.8.patch, HIVE-2646.D2133.9.patch, 
> HIVE-2646.D2883.1.patch, HIVE-2646.D2883.2.patch, HIVE-2646.D2883.3.patch, 
> HIVE-2646.diff.txt, HIVE-2646_LANG.patch
>
>
> The current Hive Ivy dependency logic for its Hadoop dependencies is 
> problematic - depending on the tarball and extracting the jars from there, 
> rather than depending on the jars directly. It'd be great if this was fixed 
> to actually have the jar dependencies defined directly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs

[Andrew Bayer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Bayer updated HIVE-2646:
---

Attachment: (was: BIGTOP-649.diff.txt)

> Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
> 
>
> Key: HIVE-2646
> URL: https://issues.apache.org/jira/browse/HIVE-2646
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Affects Versions: 0.8.0
>Reporter: Andrew Bayer
>Assignee: Andrew Bayer
>Priority: Critical
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-2646-fixtests.patch, HIVE-2646-fixtests.txt, 
> HIVE-2646.D2133.1.patch, HIVE-2646.D2133.10.patch, HIVE-2646.D2133.11.patch, 
> HIVE-2646.D2133.12.patch, HIVE-2646.D2133.13.patch, HIVE-2646.D2133.14.patch, 
> HIVE-2646.D2133.15.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, 
> HIVE-2646.D2133.4.patch, HIVE-2646.D2133.5.patch, HIVE-2646.D2133.6.patch, 
> HIVE-2646.D2133.7.patch, HIVE-2646.D2133.8.patch, HIVE-2646.D2133.9.patch, 
> HIVE-2646.D2883.1.patch, HIVE-2646.D2883.2.patch, HIVE-2646.D2883.3.patch, 
> HIVE-2646.diff.txt, HIVE-2646_LANG.patch
>
>
> The current Hive Ivy dependency logic for its Hadoop dependencies is 
> problematic - depending on the tarball and extracting the jars from there, 
> rather than depending on the jars directly. It'd be great if this was fixed 
> to actually have the jar dependencies defined directly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3152) Disallow certain character patterns in partition names

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3152:
-

Status: Open  (was: Patch Available)

@Andrew: I left some comments on phabricator. Thanks.

On a related note, please file a JIRA ticket before creating a review request, 
and make sure to update the JIRA ticket with the location of the code review 
once it's created.

> Disallow certain character patterns in partition names
> --
>
> Key: HIVE-3152
> URL: https://issues.apache.org/jira/browse/HIVE-3152
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Andrew Poland
>Assignee: Andrew Poland
>Priority: Minor
>
> New event listener to allow metastore to reject a partition name if it 
> contains undesired character patterns such as unicode and commas.
> Match pattern is implemented as a regular expression
> Modifies append_partition to call a new MetaStorePreventListener 
> implementation, PreAppendPartitionEvent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3128) use commons-compress instead of forking tar process

[Kanna Karanam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399665#comment-13399665
 ] 

Kanna Karanam commented on HIVE-3128:
-

Thanks Edward.

> use commons-compress instead of forking tar process
> ---
>
> Key: HIVE-3128
> URL: https://issues.apache.org/jira/browse/HIVE-3128
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, Query Processor
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
> Fix For: 0.10.0
>
> Attachments: HIVE-3128.1.patch.txt, HIVE-3128.2.patch.txt
>
>
> TAR tool doesn’t exist by default on windows systems so use the CAB files on 
> windows

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3128) use commons-compress instead of forking tar process

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo updated HIVE-3128:
--

   Resolution: Fixed
Fix Version/s: 0.10.0
 Assignee: Kanna Karanam
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks! Windows support here we come.

> use commons-compress instead of forking tar process
> ---
>
> Key: HIVE-3128
> URL: https://issues.apache.org/jira/browse/HIVE-3128
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, Query Processor
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
> Fix For: 0.10.0
>
> Attachments: HIVE-3128.1.patch.txt, HIVE-3128.2.patch.txt
>
>
> TAR tool doesn’t exist by default on windows systems so use the CAB files on 
> windows

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3128) use commons-compress instead of forking tar process

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo updated HIVE-3128:
--

Summary: use commons-compress instead of forking tar process  (was: Support 
the CAB format to accommodate the absence of tar command on windows)

> use commons-compress instead of forking tar process
> ---
>
> Key: HIVE-3128
> URL: https://issues.apache.org/jira/browse/HIVE-3128
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, Query Processor
>Reporter: Kanna Karanam
> Attachments: HIVE-3128.1.patch.txt, HIVE-3128.2.patch.txt
>
>
> TAR tool doesn’t exist by default on windows systems so use the CAB files on 
> windows

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3128) Support the CAB format to accommodate the absence of tar command on windows

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399651#comment-13399651
 ] 

Edward Capriolo commented on HIVE-3128:
---

+1 . Will commit

The surface time of tests swings so dramatically the performance of tar is not 
an issue. In fact a test I did ran faster with your patch then without.

{noformat}
  [junit] diff -a 
/home/edward/hive/trunk/build/ql/test/logs/clientpositive/timestamp_lazy.q.out 
/home/edward/hive/trunk/ql/src/test/results/clientpositive/timestamp_lazy.q.out
[junit] Done query: timestamp_lazy.q elapsedTime=32s


[junit] diff -a 
/home/edward/hive/trunk/build/ql/test/logs/clientpositive/timestamp_lazy.q.out 
/home/edward/hive/trunk/ql/src/test/results/clientpositive/timestamp_lazy.q.out
[junit] Done query: timestamp_lazy.q elapsedTime=38s
{noformat}

> Support the CAB format to accommodate the absence of tar command on windows
> ---
>
> Key: HIVE-3128
> URL: https://issues.apache.org/jira/browse/HIVE-3128
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, Query Processor
>Reporter: Kanna Karanam
> Attachments: HIVE-3128.1.patch.txt, HIVE-3128.2.patch.txt
>
>
> TAR tool doesn’t exist by default on windows systems so use the CAB files on 
> windows

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3128) Support the CAB format to accommodate the absence of tar command on windows

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399620#comment-13399620
 ] 

Edward Capriolo commented on HIVE-3128:
---

The unit tests usually run multiple hours, if this patch adds less then 5 
minutes this is not a big deal.

> Support the CAB format to accommodate the absence of tar command on windows
> ---
>
> Key: HIVE-3128
> URL: https://issues.apache.org/jira/browse/HIVE-3128
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, Query Processor
>Reporter: Kanna Karanam
> Attachments: HIVE-3128.1.patch.txt, HIVE-3128.2.patch.txt
>
>
> TAR tool doesn’t exist by default on windows systems so use the CAB files on 
> windows

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3166) The Hive JDBC driver should accept hive conf and hive variables via connection URL

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-3166:
--

Fix Version/s: 0.10.0
Affects Version/s: 0.9.0
 Hadoop Flags: Reviewed
   Status: Patch Available  (was: Open)

> The Hive JDBC driver should accept hive conf and hive variables via 
> connection URL
> --
>
> Key: HIVE-3166
> URL: https://issues.apache.org/jira/browse/HIVE-3166
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.9.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 0.10.0
>
> Attachments: HIVE-3166-3.patch
>
>
> The JDBC driver supports running embedded hive. The Hive CLI can accept 
> configuration and hive settings on command line that can be passed down. But 
> the JDBC driver currently doesn't support this.
> Its also required for SQLLine CLI support since that is a JDBC application. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3146) Support external hive tables whose data are stored in Azure blob store/Azure Storage Volumes (ASV)

[Kanna Karanam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399600#comment-13399600
 ] 

Kanna Karanam commented on HIVE-3146:
-

@Edward, @JQ – If I understand correctly, the suggestion is
1)Create a new setting in the hive-conf.xml
2)Set the default value to the existing values in hiveConf.java & hive-conf.xml 
template.
3)Customer can override this setting if they want to work with a different 
storage system.

I personally don’t see so many changes to this but if you strongly feel that it 
has to be configurable then please let me know. I will update the patch and 
send it for review.


> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)
> --
>
> Key: HIVE-3146
> URL: https://issues.apache.org/jira/browse/HIVE-3146
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Affects Versions: 0.10.0
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3146.1.patch.txt
>
>
> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3166) The Hive JDBC driver should accept hive conf and hive variables via connection URL

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-3166:
--

Attachment: HIVE-3166-3.patch

Attaching the reviewed patch

> The Hive JDBC driver should accept hive conf and hive variables via 
> connection URL
> --
>
> Key: HIVE-3166
> URL: https://issues.apache.org/jira/browse/HIVE-3166
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.9.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 0.10.0
>
> Attachments: HIVE-3166-3.patch
>
>
> The JDBC driver supports running embedded hive. The Hive CLI can accept 
> configuration and hive settings on command line that can be passed down. But 
> the JDBC driver currently doesn't support this.
> Its also required for SQLLine CLI support since that is a JDBC application. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3166) The Hive JDBC driver should accept hive conf and hive variables via connection URL

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399597#comment-13399597
 ] 

Carl Steinbach commented on HIVE-3166:
--

+1

@Prasad: Please attach the most recent version of the patch from reviewboard. I 
will test and commit. Thanks.

> The Hive JDBC driver should accept hive conf and hive variables via 
> connection URL
> --
>
> Key: HIVE-3166
> URL: https://issues.apache.org/jira/browse/HIVE-3166
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
>
> The JDBC driver supports running embedded hive. The Hive CLI can accept 
> configuration and hive settings on command line that can be passed down. But 
> the JDBC driver currently doesn't support this.
> Its also required for SQLLine CLI support since that is a JDBC application. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3135) add an option in ptest to run on a single machine

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3135:


Affects Version/s: 0.10.0

> add an option in ptest to run on a single machine
> -
>
> Key: HIVE-3135
> URL: https://issues.apache.org/jira/browse/HIVE-3135
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Affects Versions: 0.10.0
>Reporter: Namit Jain
>Assignee: Namit Jain
> Fix For: 0.10.0
>
>
> There is no need for any sudo in that case

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3146) Support external hive tables whose data are stored in Azure blob store/Azure Storage Volumes (ASV)

[Kanna Karanam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399586#comment-13399586
 ] 

Kanna Karanam commented on HIVE-3146:
-

please find the code review request at https://reviews.apache.org/r/5530/


> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)
> --
>
> Key: HIVE-3146
> URL: https://issues.apache.org/jira/browse/HIVE-3146
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Affects Versions: 0.10.0
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3146.1.patch.txt
>
>
> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3152) Disallow certain character patterns in partition names

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399581#comment-13399581
 ] 

Carl Steinbach commented on HIVE-3152:
--

I'm looking at this now.

> Disallow certain character patterns in partition names
> --
>
> Key: HIVE-3152
> URL: https://issues.apache.org/jira/browse/HIVE-3152
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Andrew Poland
>Assignee: Andrew Poland
>Priority: Minor
>
> New event listener to allow metastore to reject a partition name if it 
> contains undesired character patterns such as unicode and commas.
> Match pattern is implemented as a regular expression
> Modifies append_partition to call a new MetaStorePreventListener 
> implementation, PreAppendPartitionEvent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3135) add an option in ptest to run on a single machine

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3135:
-

Fix Version/s: 0.10.0

@Kevin: Please set the fix version.

> add an option in ptest to run on a single machine
> -
>
> Key: HIVE-3135
> URL: https://issues.apache.org/jira/browse/HIVE-3135
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Reporter: Namit Jain
>Assignee: Namit Jain
> Fix For: 0.10.0
>
>
> There is no need for any sudo in that case

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3146) Support external hive tables whose data are stored in Azure blob store/Azure Storage Volumes (ASV)

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3146:
-

Status: Open  (was: Patch Available)

@Kanna: please create a review request on reviews.apache.org. Thanks.

> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)
> --
>
> Key: HIVE-3146
> URL: https://issues.apache.org/jira/browse/HIVE-3146
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Affects Versions: 0.10.0
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3146.1.patch.txt
>
>
> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2955) Queries consists of metadata-only-query returns always empty value

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2955:
-

  Component/s: Query Processor
Fix Version/s: 0.10.0

@Namit: Please remember to set the fix version field when you mark a ticket as 
resolved. Thanks.

> Queries consists of metadata-only-query returns always empty value
> --
>
> Key: HIVE-2955
> URL: https://issues.apache.org/jira/browse/HIVE-2955
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.10.0
>
> Attachments: HIVE-2955.D2817.1.patch
>
>
> For partitioned table, simple query on partition column returns always null 
> or empty value, for example,
> {code}
> create table emppart(empno int, ename string) partitioned by (deptno int);
> .. load partitions..
> select distinct deptno from emppart; // empty
> select min(deptno), max(deptno) from emppart;  // NULL and NULL
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3152) Disallow certain character patterns in partition names

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399555#comment-13399555
 ] 

Kevin Wilfong commented on HIVE-3152:
-

I'm +1 on it and running tests.

I will give it some time though before committing to give others a chance to 
look.

> Disallow certain character patterns in partition names
> --
>
> Key: HIVE-3152
> URL: https://issues.apache.org/jira/browse/HIVE-3152
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Andrew Poland
>Assignee: Andrew Poland
>Priority: Minor
>
> New event listener to allow metastore to reject a partition name if it 
> contains undesired character patterns such as unicode and commas.
> Match pattern is implemented as a regular expression
> Modifies append_partition to call a new MetaStorePreventListener 
> implementation, PreAppendPartitionEvent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3152) Disallow certain character patterns in partition names

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3152:


Assignee: Andrew Poland
  Status: Patch Available  (was: Open)

> Disallow certain character patterns in partition names
> --
>
> Key: HIVE-3152
> URL: https://issues.apache.org/jira/browse/HIVE-3152
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Andrew Poland
>Assignee: Andrew Poland
>Priority: Minor
>
> New event listener to allow metastore to reject a partition name if it 
> contains undesired character patterns such as unicode and commas.
> Match pattern is implemented as a regular expression
> Modifies append_partition to call a new MetaStorePreventListener 
> implementation, PreAppendPartitionEvent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3152) Disallow certain character patterns in partition names

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399552#comment-13399552
 ] 

Kevin Wilfong commented on HIVE-3152:
-

Andrew Poland submitted a diff for this change:
https://reviews.facebook.net/D3711

> Disallow certain character patterns in partition names
> --
>
> Key: HIVE-3152
> URL: https://issues.apache.org/jira/browse/HIVE-3152
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Andrew Poland
>Priority: Minor
>
> New event listener to allow metastore to reject a partition name if it 
> contains undesired character patterns such as unicode and commas.
> Match pattern is implemented as a regular expression
> Modifies append_partition to call a new MetaStorePreventListener 
> implementation, PreAppendPartitionEvent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-h0.21 - Build # 1504 - Fixed

[Apache Jenkins Server]

Changes for Build #1503
[cws] HIVE-3161. A minor test update
(Namit Jain via Carl Steinbach)

Summary: The correct long term fix is HIVE-3160

Test Plan: manual

Differential Revision: https://reviews.facebook.net/D3723


Changes for Build #1504
[namit] HIVE-2955 Queries consists of metadata-only-query returns always empty 
value
(Navis via namit)




All tests passed

The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1504)

Status: Fixed

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1504/ to 
view the results.

[jira] [Commented] (HIVE-2955) Queries consists of metadata-only-query returns always empty value

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399537#comment-13399537
 ] 

Hudson commented on HIVE-2955:
--

Integrated in Hive-trunk-h0.21 #1504 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/1504/])
HIVE-2955 Queries consists of metadata-only-query returns always empty value
(Navis via namit) (Revision 1352771)

 Result = SUCCESS
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352771
Files : 
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java
* /hive/trunk/ql/src/test/queries/clientpositive/metadataonly1.q
* /hive/trunk/ql/src/test/results/clientpositive/metadataonly1.q.out


> Queries consists of metadata-only-query returns always empty value
> --
>
> Key: HIVE-2955
> URL: https://issues.apache.org/jira/browse/HIVE-2955
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-2955.D2817.1.patch
>
>
> For partitioned table, simple query on partition column returns always null 
> or empty value, for example,
> {code}
> create table emppart(empno int, ename string) partitioned by (deptno int);
> .. load partitions..
> select distinct deptno from emppart; // empty
> select min(deptno), max(deptno) from emppart;  // NULL and NULL
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3146) Support external hive tables whose data are stored in Azure blob store/Azure Storage Volumes (ASV)

[JQ Hadoop (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399449#comment-13399449
 ] 

JQ Hadoop commented on HIVE-3146:
-

I think make it a configuration setting will be more flexible.

> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)
> --
>
> Key: HIVE-3146
> URL: https://issues.apache.org/jira/browse/HIVE-3146
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Affects Versions: 0.10.0
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3146.1.patch.txt
>
>
> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HIVE-3178) retry not honored in RetryingRawMetastore

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong reassigned HIVE-3178:
---

Assignee: Namit Jain

> retry not honored in RetryingRawMetastore
> -
>
> Key: HIVE-3178
> URL: https://issues.apache.org/jira/browse/HIVE-3178
> Project: Hive
>  Issue Type: Bug
>Reporter: Namit Jain
>Assignee: Namit Jain
> Attachments: hive.3178.1.patch
>
>
> The retrymetastore catches JDOException, but they are always wrapped by 
> reflection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3135) add an option in ptest to run on a single machine

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3135:


Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed, thanks Namit.

> add an option in ptest to run on a single machine
> -
>
> Key: HIVE-3135
> URL: https://issues.apache.org/jira/browse/HIVE-3135
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> There is no need for any sudo in that case

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3068) Add ability to export table metadata as JSON on table drop

[Andrew Chalfant (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Chalfant updated HIVE-3068:
--

Issue Type: New Feature  (was: Bug)

> Add ability to export table metadata as JSON on table drop
> --
>
> Key: HIVE-3068
> URL: https://issues.apache.org/jira/browse/HIVE-3068
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Serializers/Deserializers
>Reporter: Andrew Chalfant
>Priority: Minor
>  Labels: features, newbie
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When a table is dropped, the contents go to the users trash but the metadata 
> is lost. It would be super neat to be able to save the metadata as well so 
> that tables could be trivially re-instantiated via thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3148) LOCATION clause is not honored when adding multiple partitions

[JQ Hadoop (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399431#comment-13399431
 ] 

JQ Hadoop commented on HIVE-3148:
-

@Carl @Shengsheng: This looks a useful fix - we have some of the same issues 
with our customers too; can anyone help confirm whether this is a valid patch? 
Thanks.

> LOCATION clause is not honored when adding multiple partitions
> --
>
> Key: HIVE-3148
> URL: https://issues.apache.org/jira/browse/HIVE-3148
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Query Processor
>Affects Versions: 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0
>Reporter: Carl Steinbach
>  Labels: patch
> Attachments: 3148.for0.9.0.patch, HIVE-3148.1.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #55

[Apache Jenkins Server]

See 


--
[...truncated 10109 lines...]
 [echo] Project: odbc
 [copy] Warning: 

 does not exist.

ivy-resolve-test:
 [echo] Project: odbc

ivy-retrieve-test:
 [echo] Project: odbc

compile-test:
 [echo] Project: odbc

create-dirs:
 [echo] Project: serde
 [copy] Warning: 

 does not exist.

init:
 [echo] Project: serde

ivy-init-settings:
 [echo] Project: serde

ivy-resolve:
 [echo] Project: serde
[ivy:resolve] :: loading settings :: file = 

[ivy:report] Processing 

 to 


ivy-retrieve:
 [echo] Project: serde

dynamic-serde:

compile:
 [echo] Project: serde

ivy-resolve-test:
 [echo] Project: serde

ivy-retrieve-test:
 [echo] Project: serde

compile-test:
 [echo] Project: serde
[javac] Compiling 26 source files to 

[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.

create-dirs:
 [echo] Project: service
 [copy] Warning: 

 does not exist.

init:
 [echo] Project: service

ivy-init-settings:
 [echo] Project: service

ivy-resolve:
 [echo] Project: service
[ivy:resolve] :: loading settings :: file = 

[ivy:report] Processing 

 to 


ivy-retrieve:
 [echo] Project: service

compile:
 [echo] Project: service

ivy-resolve-test:
 [echo] Project: service

ivy-retrieve-test:
 [echo] Project: service

compile-test:
 [echo] Project: service
[javac] Compiling 2 source files to 


test:
 [echo] Project: hive

test-shims:
 [echo] Project: hive

test-conditions:
 [echo] Project: shims

gen-test:
 [echo] Project: shims

create-dirs:
 [echo] Project: shims
 [copy] Warning: 

 does not exist.

init:
 [echo] Project: shims

ivy-init-settings:
 [echo] Project: shims

ivy-resolve:
 [echo] Project: shims
[ivy:resolve] :: loading settings :: file = 

[ivy:report] Processing 

 to 


ivy-retrieve:
 [echo] Project: shims

compile:
 [echo] Project: shims
 [echo] Building shims 0.20

build_shims:
 [echo] Project: shims
 [echo] Compiling 

 against hadoop 0.20.2 
(

ivy-init-settings:
 [echo] Project: shims

ivy-resolve-hadoop-shim:
 [echo] Project: shims
[ivy:resolve] :: loading settings :: file = 


ivy-retrieve-hadoop-shim:
 [echo] Project: shims
 [echo] Building shims 0.20S

build_shims:
 [echo] Project: shims
 [echo] Compiling 

Re: Concurrency in hive

[Edward Capriolo]

Almost all operations in hive can exploit map reduce for parallelism.
(isnt not really done on the thread level) essentially if you run a
hive job and there is multiple mappers or reducers it was parallelism.

On Fri, Jun 22, 2012 at 5:14 AM, Jayanth Muthya  wrote:
> Thanks or clarifying, I'll look into it too and see if I can find anything.
>
> -Jayanth
>
> On Thu, Jun 21, 2012 at 10:47 PM, Jerome Banks  wrote:
>
>> set hive.exec.parallel=true;
>>
>> This will run Hive jobs in parallel, if they are able to do so.
>>
>> As for multi-threading in the actual job itself, I don't think so, but I'm
>> not sure. The query planner will merge steps together, in order to try to
>> minimize the number of MR jobs needed to run a query, but I think those are
>> chained together in a single thread, both on the mapper and reduce.
>>
>> When I was at Quantcast, we had some multi-threading in the mapper ands
>> reducers, to try to increase throughput, by utilizing the CPU when the job
>> would otherwise be blocked on IO.  This helps out, if your IO is very slow,
>> but if the IO no longer becomes a bottleneck, then you spend a lot of time
>> context-switching, and it no longer efficient.
>>
>> Interesting question, I'll look into it some more. Let me know if you find
>> out anything.
>>
>> -- jerome
>>
>> On Thu, Jun 21, 2012 at 1:16 AM, Jayanth Muthya > >wrote:
>>
>> > Hi,
>> > I was looking into some of the source code for hive. And had a few
>> > questions regarding parallelism in hive. Can a map task in
>> > hive exploit parallelism and run multiple threads? If it can do that,
>> does
>> > it do it by default? or does a user have to configure the settings?
>> > This question seems really basic, I just started looking into
>> hadoop/hive.
>> > Thanks in advance!
>> >
>> > -Jay
>> >
>>

[jira] [Commented] (HIVE-3179) HBase Handler doesn't handle NULLs properly

[Lars Francke (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399322#comment-13399322
 ] 

Lars Francke commented on HIVE-3179:


We could add a second boolean array to go with {{fieldsInited}} that's called 
{{fieldsNull}} that caches those fields. Not sure if that's needed though.

Thanks to my colleague Oliver Meyn who actually looked at the code and found 
the fix, I only packaged it up and added the unit test.



> HBase Handler doesn't handle NULLs properly
> ---
>
> Key: HIVE-3179
> URL: https://issues.apache.org/jira/browse/HIVE-3179
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Affects Versions: 0.9.0
>Reporter: Lars Francke
>Priority: Critical
> Attachments: HIVE-3179.1.patch
>
>
> We found a quite severe issue in the HBase Handler which actually means that 
> Hive potentially returns incorrect data if a column has NULL values in HBase 
> (which means the cell doesn't even exist)
> In HBase Shell:
> {noformat}
> create 'hive_hbase_test', 'test'
> put 'hive_hbase_test', '1', 'test:c1', 'c1-1'
> put 'hive_hbase_test', '1', 'test:c2', 'c2-1'
> put 'hive_hbase_test', '1', 'test:c3', 'c3-1'
> put 'hive_hbase_test', '2', 'test:c1', 'c1-2'
> {noformat}
> In Hive:
> {noformat}
> DROP TABLE IF EXISTS hive_hbase_test;
> CREATE EXTERNAL TABLE hive_hbase_test (
>   id int,
>   c1 string,
>   c2 string,
>   c3 string
> )
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" =
> ":key#s,test:c1#s,test:c2#s,test:c3#s")
> TBLPROPERTIES("hbase.table.name" = "hive_hbase_test");
> hive> select * from hive_hbase_test;
> OK
> 1 c1-1c2-1c3-1
> 2 c1-2NULLNULL
> hive> select c1 from hive_hbase_test;
> c1-1
> c1-2
> hive> select c1, c2 from hive_hbase_test;
> c1-1  c2-1
> c1-2  NULL
> {noformat}
> So far everything is correct but now:
> {noformat}
> hive> select c1, c2, c2 from hive_hbase_test;
> c1-1  c2-1c2-1
> c1-2  NULLc2-1
> {noformat}
> Selecting c2 twice works the first time but the second time we
> actually get the value from the previous row.
> {noformat}
> hive> select c1, c3, c2, c2, c3, c3, c1 from hive_hbase_test;
> c1-1  c3-1c2-1c2-1c3-1c3-1c1-1
> c1-2  NULLNULLc2-1c3-1c3-1c1-2
> {noformat}
> We've narrowed this down to an early initialization of 
> {{fieldsInited\[fieldID] = true}} in {{LazyHBaseRow#uncheckedGetField}} and 
> we'll try to provide a patch which surely needs review.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3179) HBase Handler doesn't handle NULLs properly

[Lars Francke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Francke updated HIVE-3179:
---

Attachment: HIVE-3179.1.patch

The attached patch fixes the problem as well as changes a unit test that 
actually tests this behavior. The unit test fails if our fix to 
{{LazyHBaseRow}} is not applied.

We're not sure if this is the best way to fix this problem as it circumvents 
the optimization being done by the fieldsInited field. Ideally instead of 
returning null on an empty HBase cell this would insert some kind of marker but 
adding an empty ByteArrayRef is not interpreted as NULL but as an empty value 
(which makes sense).

In short: This fixes the bug at the cost of some performance for NULL 
(non-existing) fields in HBase.

> HBase Handler doesn't handle NULLs properly
> ---
>
> Key: HIVE-3179
> URL: https://issues.apache.org/jira/browse/HIVE-3179
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Affects Versions: 0.9.0
>Reporter: Lars Francke
>Priority: Critical
> Attachments: HIVE-3179.1.patch
>
>
> We found a quite severe issue in the HBase Handler which actually means that 
> Hive potentially returns incorrect data if a column has NULL values in HBase 
> (which means the cell doesn't even exist)
> In HBase Shell:
> {noformat}
> create 'hive_hbase_test', 'test'
> put 'hive_hbase_test', '1', 'test:c1', 'c1-1'
> put 'hive_hbase_test', '1', 'test:c2', 'c2-1'
> put 'hive_hbase_test', '1', 'test:c3', 'c3-1'
> put 'hive_hbase_test', '2', 'test:c1', 'c1-2'
> {noformat}
> In Hive:
> {noformat}
> DROP TABLE IF EXISTS hive_hbase_test;
> CREATE EXTERNAL TABLE hive_hbase_test (
>   id int,
>   c1 string,
>   c2 string,
>   c3 string
> )
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" =
> ":key#s,test:c1#s,test:c2#s,test:c3#s")
> TBLPROPERTIES("hbase.table.name" = "hive_hbase_test");
> hive> select * from hive_hbase_test;
> OK
> 1 c1-1c2-1c3-1
> 2 c1-2NULLNULL
> hive> select c1 from hive_hbase_test;
> c1-1
> c1-2
> hive> select c1, c2 from hive_hbase_test;
> c1-1  c2-1
> c1-2  NULL
> {noformat}
> So far everything is correct but now:
> {noformat}
> hive> select c1, c2, c2 from hive_hbase_test;
> c1-1  c2-1c2-1
> c1-2  NULLc2-1
> {noformat}
> Selecting c2 twice works the first time but the second time we
> actually get the value from the previous row.
> {noformat}
> hive> select c1, c3, c2, c2, c3, c3, c1 from hive_hbase_test;
> c1-1  c3-1c2-1c2-1c3-1c3-1c1-1
> c1-2  NULLNULLc2-1c3-1c3-1c1-2
> {noformat}
> We've narrowed this down to an early initialization of 
> {{fieldsInited\[fieldID] = true}} in {{LazyHBaseRow#uncheckedGetField}} and 
> we'll try to provide a patch which surely needs review.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

count of distinct FROM multiple columns

[MIS]

Hi All,

I have a table in Hive as below:

dummy {
col1 STRING,
col2 INT,
col3 INT
}

And in that there is some sample data as :

*col1  col2 col3 *
ABC45
XYZ12
ABC13
ABC51
XYZ31

What should be my query so as to get the below result:

*ABC 4
XYZ 3*

Basically I'm trying to get a count of distinct elements from *col2 and
col3 combined* and group them against col1.

I tried with a query as:

select col1, count(distinct col2, col3) from dummy group by col1

But didn't get the expected output.

Can anybody point me in the correct direction and suggest a correct query.

Thanks.

[jira] [Updated] (HIVE-3179) HBase Handler doesn't handle NULLs properly

[Lars Francke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Francke updated HIVE-3179:
---

Description: 
We found a quite severe issue in the HBase Handler which actually means that 
Hive potentially returns incorrect data if a column has NULL values in HBase 
(which means the cell doesn't even exist)

In HBase Shell:

{noformat}
create 'hive_hbase_test', 'test'
put 'hive_hbase_test', '1', 'test:c1', 'c1-1'
put 'hive_hbase_test', '1', 'test:c2', 'c2-1'
put 'hive_hbase_test', '1', 'test:c3', 'c3-1'
put 'hive_hbase_test', '2', 'test:c1', 'c1-2'
{noformat}

In Hive:

{noformat}
DROP TABLE IF EXISTS hive_hbase_test;
CREATE EXTERNAL TABLE hive_hbase_test (
  id int,
  c1 string,
  c2 string,
  c3 string
)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" =
":key#s,test:c1#s,test:c2#s,test:c3#s")
TBLPROPERTIES("hbase.table.name" = "hive_hbase_test");

hive> select * from hive_hbase_test;
OK
1   c1-1c2-1c3-1
2   c1-2NULLNULL

hive> select c1 from hive_hbase_test;
c1-1
c1-2

hive> select c1, c2 from hive_hbase_test;
c1-1c2-1
c1-2NULL
{noformat}

So far everything is correct but now:

{noformat}
hive> select c1, c2, c2 from hive_hbase_test;
c1-1c2-1c2-1
c1-2NULLc2-1
{noformat}

Selecting c2 twice works the first time but the second time we
actually get the value from the previous row.

{noformat}
hive> select c1, c3, c2, c2, c3, c3, c1 from hive_hbase_test;
c1-1c3-1c2-1c2-1c3-1c3-1c1-1
c1-2NULLNULLc2-1c3-1c3-1c1-2
{noformat}

We've narrowed this down to an early initialization of {{fieldsInited\[fieldID] 
= true}} in {{LazyHBaseRow#uncheckedGetField}} and we'll try to provide a patch 
which surely needs review.

  was:
We found a quite severe issue in the HBase Handler which actually means that 
Hive potentially returns incorrect data if a column has NULL values in HBase 
(which means the cell doesn't even exist)

In HBase Shell:

{noformat}
create 'hive_hbase_test', 'test'
put 'hive_hbase_test', '1', 'test:c1', 'c1-1'
put 'hive_hbase_test', '1', 'test:c2', 'c2-1'
put 'hive_hbase_test', '1', 'test:c3', 'c3-1'
put 'hive_hbase_test', '2', 'test:c1', 'c1-2'
{noformat}

In Hive:

{noformat}
DROP TABLE IF EXISTS hive_hbase_test;
CREATE EXTERNAL TABLE hive_hbase_test (
  id int,
  c1 string,
  c2 string,
  c3 string
)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" =
":key#s,test:c1#s,test:c2#s,test:c3#s")
TBLPROPERTIES("hbase.table.name" = "hive_hbase_test");

hive> select * from hive_hbase_test;
OK
1   c1-1c2-1c3-1
2   c1-2NULLNULL

hive> select c1 from hive_hbase_test;
c1-1
c1-2

hive> select c1, c2 from hive_hbase_test;
c1-1c2-1
c1-2NULL
{noformat}

So far everything is correct but now:

{noformat}
hive> select c1, c2, c2 from hive_hbase_test;
c1-1c2-1c2-1
c1-2NULLc2-1
{noformat}

Selecting c2 twice works the first time but the second time we
actually get the value from the previous row.

{noformat}
hive> select c1, c3, c2, c2, c3, c3, c1 from hive_hbase_test;
c1-1c3-1c2-1c2-1c3-1c3-1c1-1
c1-2NULLNULLc2-1c3-1c3-1c1-2
{noformat}

We've narrowed this down to an early initialization of {{fieldsInited[fieldID] 
= true;}} in {{LazyHBaseRow#uncheckedGetField}} and we'll try to provide a 
patch which surely needs review.


> HBase Handler doesn't handle NULLs properly
> ---
>
> Key: HIVE-3179
> URL: https://issues.apache.org/jira/browse/HIVE-3179
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Affects Versions: 0.9.0
>Reporter: Lars Francke
>Priority: Critical
>
> We found a quite severe issue in the HBase Handler which actually means that 
> Hive potentially returns incorrect data if a column has NULL values in HBase 
> (which means the cell doesn't even exist)
> In HBase Shell:
> {noformat}
> create 'hive_hbase_test', 'test'
> put 'hive_hbase_test', '1', 'test:c1', 'c1-1'
> put 'hive_hbase_test', '1', 'test:c2', 'c2-1'
> put 'hive_hbase_test', '1', 'test:c3', 'c3-1'
> put 'hive_hbase_test', '2', 'test:c1', 'c1-2'
> {noformat}
> In Hive:
> {noformat}
> DROP TABLE IF EXISTS hive_hbase_test;
> CREATE EXTERNAL TABLE hive_hbase_test (
>   id int,
>   c1 string,
>   c2 string,
>   c3 string
> )
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" =
> ":key#s,test:c1#s,test:c2#s,test:c3#s")
> TBLPROPERTIES("hbase.table.name" = "hive_hbase_test");
> hive> select * from hive_hbase_test;
> OK
> 1 c1-1c2-1c3-1
> 2 c1-2NULLNULL
> hive> select c1 from hive_hbase_test;
> c1-1
> c1-2
> hive> select c1, c2

[jira] [Created] (HIVE-3179) HBase Handler doesn't handle NULLs properly

[Lars Francke (JIRA)]

Lars Francke created HIVE-3179:
--

 Summary: HBase Handler doesn't handle NULLs properly
 Key: HIVE-3179
 URL: https://issues.apache.org/jira/browse/HIVE-3179
 Project: Hive
  Issue Type: Bug
  Components: HBase Handler
Affects Versions: 0.9.0
Reporter: Lars Francke
Priority: Critical


We found a quite severe issue in the HBase Handler which actually means that 
Hive potentially returns incorrect data if a column has NULL values in HBase 
(which means the cell doesn't even exist)

In HBase Shell:

{noformat}
create 'hive_hbase_test', 'test'
put 'hive_hbase_test', '1', 'test:c1', 'c1-1'
put 'hive_hbase_test', '1', 'test:c2', 'c2-1'
put 'hive_hbase_test', '1', 'test:c3', 'c3-1'
put 'hive_hbase_test', '2', 'test:c1', 'c1-2'
{noformat}

In Hive:

{noformat}
DROP TABLE IF EXISTS hive_hbase_test;
CREATE EXTERNAL TABLE hive_hbase_test (
  id int,
  c1 string,
  c2 string,
  c3 string
)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" =
":key#s,test:c1#s,test:c2#s,test:c3#s")
TBLPROPERTIES("hbase.table.name" = "hive_hbase_test");

hive> select * from hive_hbase_test;
OK
1   c1-1c2-1c3-1
2   c1-2NULLNULL

hive> select c1 from hive_hbase_test;
c1-1
c1-2

hive> select c1, c2 from hive_hbase_test;
c1-1c2-1
c1-2NULL
{noformat}

So far everything is correct but now:

{noformat}
hive> select c1, c2, c2 from hive_hbase_test;
c1-1c2-1c2-1
c1-2NULLc2-1
{noformat}

Selecting c2 twice works the first time but the second time we
actually get the value from the previous row.

{noformat}
hive> select c1, c3, c2, c2, c3, c3, c1 from hive_hbase_test;
c1-1c3-1c2-1c2-1c3-1c3-1c1-1
c1-2NULLNULLc2-1c3-1c3-1c1-2
{noformat}

We've narrowed this down to an early initialization of {{fieldsInited[fieldID] 
= true;}} in {{LazyHBaseRow#uncheckedGetField}} and we'll try to provide a 
patch which surely needs review.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3173) implement getTypeInfo database metadata method

[N Campbell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

N Campbell updated HIVE-3173:
-

Issue Type: Bug  (was: Improvement)

> implement getTypeInfo database metadata method 
> ---
>
> Key: HIVE-3173
> URL: https://issues.apache.org/jira/browse/HIVE-3173
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.8.1
>Reporter: N Campbell
>
> The JDBC driver does not implement the database metadata method getTypeInfo. 
> Hence, an application cannot dynamically determine the available type 
> information and associated properties. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3174) should not be throwing exceptions for many JDBC Database metadata properties

[N Campbell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

N Campbell updated HIVE-3174:
-

Issue Type: Bug  (was: Improvement)

> should not be throwing exceptions for many JDBC Database metadata properties
> 
>
> Key: HIVE-3174
> URL: https://issues.apache.org/jira/browse/HIVE-3174
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.8.1
>Reporter: N Campbell
>
> When a JDBC application attempts to access numerous database metadata 
> properties it will frequently encounter the driver throwing an exception. It 
> should not throw exceptions and should return valid values per the JDBC 
> specification etc. If you were to use an open source application such as 
> SQLSquirrel and attempt to view the Metadata tab a lot of properties will be 
> shown as  which is them handling the exception vs the expected 
> values such as False or True etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3175) JDBC driver should implement returning values for getTableName, getSchemaName etc for ResultSet metadata

[N Campbell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

N Campbell updated HIVE-3175:
-

Issue Type: Bug  (was: Improvement)

> JDBC driver should implement returning values for getTableName, getSchemaName 
> etc for ResultSet metadata
> 
>
> Key: HIVE-3175
> URL: https://issues.apache.org/jira/browse/HIVE-3175
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.8.1
>Reporter: N Campbell
>
> ResultSet metadata methods getTableName, getSchemaName, getCatalogName etc 
> should be implemented per the JDBC 3/4 specification

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-1643) support range scans and non-key columns in HBase filter pushdown

[bharath v (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

bharath v updated HIVE-1643:


Attachment: HIVE-1643.patch

> support range scans and non-key columns in HBase filter pushdown
> 
>
> Key: HIVE-1643
> URL: https://issues.apache.org/jira/browse/HIVE-1643
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Affects Versions: 0.9.0
>Reporter: John Sichi
>Assignee: bharath v
>  Labels: patch
> Attachments: HIVE-1643.patch, hbase_handler.patch
>
>
> HIVE-1226 added support for WHERE rowkey=3.  We would like to support WHERE 
> rowkey BETWEEN 10 and 20, as well as predicates on non-rowkeys (plus 
> conjunctions etc).  Non-rowkey conditions can't be used to filter out entire 
> ranges, but they can be used to push the per-row filter processing as far 
> down as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-1643) support range scans and non-key columns in HBase filter pushdown

[bharath v (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

bharath v updated HIVE-1643:


 Assignee: bharath v  (was: Vaibhav Aggarwal)
   Labels: patch  (was: )
Affects Version/s: (was: 0.7.0)
   0.9.0
   Status: Patch Available  (was: Open)

This patch converts *non row-key* predicates into corresponding hbase filters 
and adds them to scan objects to save network IO. Current trunk does filtering 
after TableScan operator.Tested it by observing reduction in number of rows 
processed by TableScanOperator. 

> support range scans and non-key columns in HBase filter pushdown
> 
>
> Key: HIVE-1643
> URL: https://issues.apache.org/jira/browse/HIVE-1643
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Affects Versions: 0.9.0
>Reporter: John Sichi
>Assignee: bharath v
>  Labels: patch
> Attachments: HIVE-1643.patch, hbase_handler.patch
>
>
> HIVE-1226 added support for WHERE rowkey=3.  We would like to support WHERE 
> rowkey BETWEEN 10 and 20, as well as predicates on non-rowkeys (plus 
> conjunctions etc).  Non-rowkey conditions can't be used to filter out entire 
> ranges, but they can be used to push the per-row filter processing as far 
> down as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Concurrency in hive

[Jayanth Muthya]

Thanks or clarifying, I'll look into it too and see if I can find anything.

-Jayanth

On Thu, Jun 21, 2012 at 10:47 PM, Jerome Banks  wrote:

> set hive.exec.parallel=true;
>
> This will run Hive jobs in parallel, if they are able to do so.
>
> As for multi-threading in the actual job itself, I don't think so, but I'm
> not sure. The query planner will merge steps together, in order to try to
> minimize the number of MR jobs needed to run a query, but I think those are
> chained together in a single thread, both on the mapper and reduce.
>
> When I was at Quantcast, we had some multi-threading in the mapper ands
> reducers, to try to increase throughput, by utilizing the CPU when the job
> would otherwise be blocked on IO.  This helps out, if your IO is very slow,
> but if the IO no longer becomes a bottleneck, then you spend a lot of time
> context-switching, and it no longer efficient.
>
> Interesting question, I'll look into it some more. Let me know if you find
> out anything.
>
> -- jerome
>
> On Thu, Jun 21, 2012 at 1:16 AM, Jayanth Muthya  >wrote:
>
> > Hi,
> > I was looking into some of the source code for hive. And had a few
> > questions regarding parallelism in hive. Can a map task in
> > hive exploit parallelism and run multiple threads? If it can do that,
> does
> > it do it by default? or does a user have to configure the settings?
> > This question seems really basic, I just started looking into
> hadoop/hive.
> > Thanks in advance!
> >
> > -Jay
> >
>

Re: Review Request: HIVE-3166: The Hive JDBC driver should accept hive conf and hive variables via connection URL

[Prasad Mujumdar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5507/
---

(Updated June 22, 2012, 8:43 a.m.)


Review request for hive, Carl Steinbach and Carl Steinbach.


Changes
---

Updates per review comments.


Description
---

Support passing configuration and substitution variable as part of JDBC 
connection string. 
The new format of the URL is 
jdbc:hive://:/dbName?hive_conf_list#hive_var_list   , where the 
optional conf and var lists are semicolon separated = pairs. As 
before, if the host/port is not specified, it the driver runs an embedded hive.
examples -
jdbc:hive://ubuntu:11000/db2?hive.cli.conf.printheader=true;hive.exec.mode.local.auto.inputbytes.max=#stab=salesTable;icol=customerID
jdbc:hive://?hive.cli.conf.printheader=true;hive.exec.mode.local.auto.inputbytes.max=#stab=salesTable;icol=customerID

The patch include new routines to parse the URL. The conf values are added to 
HiveConf when hive is running in embedded mode otherwise they are configured 
using 'set' statement. The variable substitution is only used in case of 
embedded mode.


This addresses bug HIVE-3166.
https://issues.apache.org/jira/browse/HIVE-3166


Diffs (updated)
-

  jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 6618243 
  jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDriver.java c61425f 
  jdbc/src/java/org/apache/hadoop/hive/jdbc/Utils.java 24d5882 
  jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java f6a904f 

Diff: https://reviews.apache.org/r/5507/diff/


Testing
---

Ran JDBC tests.
Added new test case for the extended URL format.


Thanks,

Prasad Mujumdar

Re: Review Request: HIVE-3166: The Hive JDBC driver should accept hive conf and hive variables via connection URL

[Prasad Mujumdar]

> On June 22, 2012, 5:21 a.m., Carl Steinbach wrote:
> > jdbc/src/java/org/apache/hadoop/hive/jdbc/Utils.java, line 51
> > 
> >
> > Change the name to 'isEmbeddedMode'?

Done


> On June 22, 2012, 5:21 a.m., Carl Steinbach wrote:
> > jdbc/src/java/org/apache/hadoop/hive/jdbc/Utils.java, line 128
> > 
> >
> > Space between method name and parameter list.

Done


> On June 22, 2012, 5:21 a.m., Carl Steinbach wrote:
> > jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java, line 89
> > 
> >
> > Doesn't seem like the String arrays really add anything. May as well 
> > remove them.

It makes it easier to compare the then in resultset, at least the conf 
settings. Removed the hiveVar[].


> On June 22, 2012, 5:21 a.m., Carl Steinbach wrote:
> > jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java, line 210
> > 
> >
> > Might be a good idea to create a new connection handle for this test 
> > and set the configuration parameters for this URL only.

ok. 
Changed the setup() to set the additional parameters only for this test.


> On June 22, 2012, 5:21 a.m., Carl Steinbach wrote:
> > jdbc/src/java/org/apache/hadoop/hive/jdbc/Utils.java, line 156
> > 
> >
> > The apache httpcomponents library has an URLEncodedUtils.parse() method 
> > that does this. I'd prefer that we use it here since it reduces our test 
> > burden and probably does escaping.
> > 
> > 
> > http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/utils/URLEncodedUtils.html
> >

hmm ... that's going add a dependency on a new library just for that one small 
parsing. I have it to use a regex to make it even simpler.
Please take a look and let me know if you still prefer using httpcomponents


> On June 22, 2012, 5:21 a.m., Carl Steinbach wrote:
> > jdbc/src/java/org/apache/hadoop/hive/jdbc/Utils.java, line 152
> > 
> >
> > We may want to support syntax like this in the future:
> > 
> > jdbc:///;x=1?hive.a=2
> > 
> > 
> > Where x=1 is a configuration parameter for the driver as opposed to 
> > something that should get set in the HiveConf. In order to be forward 
> > compatible with the future syntax we should make sure that dbName is the 
> > substring up to the first ';'.

sure.
In fact the dbname itself is really a parameter to driver. And the driver is 
currently not switching to the db specified in the URL.
I will log a separate ticket and submit a patch (execute 'database ' after 
the connection is established).


- Prasad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5507/#review8473
---


On June 22, 2012, 2:48 a.m., Prasad Mujumdar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5507/
> ---
> 
> (Updated June 22, 2012, 2:48 a.m.)
> 
> 
> Review request for hive, Carl Steinbach and Carl Steinbach.
> 
> 
> Description
> ---
> 
> Support passing configuration and substitution variable as part of JDBC 
> connection string. 
> The new format of the URL is 
> jdbc:hive://:/dbName?hive_conf_list#hive_var_list   , where the 
> optional conf and var lists are semicolon separated = pairs. As 
> before, if the host/port is not specified, it the driver runs an embedded 
> hive.
> examples -
> jdbc:hive://ubuntu:11000/db2?hive.cli.conf.printheader=true;hive.exec.mode.local.auto.inputbytes.max=#stab=salesTable;icol=customerID
> jdbc:hive://?hive.cli.conf.printheader=true;hive.exec.mode.local.auto.inputbytes.max=#stab=salesTable;icol=customerID
> 
> The patch include new routines to parse the URL. The conf values are added to 
> HiveConf when hive is running in embedded mode otherwise they are configured 
> using 'set' statement. The variable substitution is only used in case of 
> embedded mode.
> 
> 
> This addresses bug HIVE-3166.
> https://issues.apache.org/jira/browse/HIVE-3166
> 
> 
> Diffs
> -
> 
>   jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 6618243 
>   jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDriver.java c61425f 
>   jdbc/src/java/org/apache/hadoop/hive/jdbc/Utils.java 24d5882 
>   jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java f6a904f 
> 
> Diff: https://reviews.apache.org/r/5507/diff/
> 
> 
> Testing
> ---
> 
> Ran JDBC tests.

[jira] [Updated] (HIVE-3146) Support external hive tables whose data are stored in Azure blob store/Azure Storage Volumes (ASV)

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-3146:
---

Assignee: Kanna Karanam
  Status: Patch Available  (was: Open)

> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)
> --
>
> Key: HIVE-3146
> URL: https://issues.apache.org/jira/browse/HIVE-3146
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Affects Versions: 0.10.0
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3146.1.patch.txt
>
>
> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3146) Support external hive tables whose data are stored in Azure blob store/Azure Storage Volumes (ASV)

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399181#comment-13399181
 ] 

Ashutosh Chauhan commented on HIVE-3146:


@Ed,
According to 
https://issues.apache.org/jira/browse/HIVE-1624?focusedCommentId=12914176&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12914176
 schemes were picked rather defensively. Adding ASV in there should be fine.

> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)
> --
>
> Key: HIVE-3146
> URL: https://issues.apache.org/jira/browse/HIVE-3146
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Affects Versions: 0.10.0
>Reporter: Kanna Karanam
>  Labels: Windows
> Fix For: 0.10.0
>
> Attachments: HIVE-3146.1.patch.txt
>
>
> Support external hive tables whose data are stored in Azure blob store/Azure 
> Storage Volumes (ASV)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira