[jira] [Commented] (HIVE-2822) Add JSON output to the hive ddl commands
[
https://issues.apache.org/jira/browse/HIVE-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247055#comment-13247055
]
Phabricator commented on HIVE-2822:
---
ctdean has committed the revision HIVE-2822 [jira] Add JSON output to the hive
ddl commands.
Change committed by hashutosh.
REVISION DETAIL
https://reviews.facebook.net/D2475
COMMIT
https://reviews.facebook.net/rHIVE1309666
Add JSON output to the hive ddl commands
Key: HIVE-2822
URL: https://issues.apache.org/jira/browse/HIVE-2822
Project: Hive
Issue Type: Improvement
Reporter: Chris Dean
Assignee: Chris Dean
Attachments: HIVE-2822.03-branch0-8.patch, HIVE-2822.03.patch,
HIVE-2822.03b.patch, HIVE-2822.04-branch-08.patch,
HIVE-2822.05-branch0-8-1.patch, HIVE-2822.05-branch0-8.patch,
HIVE-2822.05.patch, HIVE-2822.D2475.1.patch, HIVE-2822.D2475.2.patch,
HIVE-2822.D2475.3.patch, HIVE-2822.D2475.4.patch,
hive-json-01-branch0-8.patch, hive-json-01.patch,
hive-json-02-branch0-8.patch, hive-json-02.patch
The goal is to have an option to produce JSON output of the DDL commands that
is easily machine parseable.
For example, desc my_table currently gives
{noformat}
idbigint
user string
{noformat}
and we want to allow a json output:
{noformat}
{
columns: [
{name: id, type: bigint},
{name: user, type: string}
]
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2822) Add JSON output to the hive ddl commands
[
https://issues.apache.org/jira/browse/HIVE-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan resolved HIVE-2822.
Resolution: Fixed
Fix Version/s: 0.9.0
Committed to trunk. Thanks, Chris!
Add JSON output to the hive ddl commands
Key: HIVE-2822
URL: https://issues.apache.org/jira/browse/HIVE-2822
Project: Hive
Issue Type: Improvement
Reporter: Chris Dean
Assignee: Chris Dean
Fix For: 0.9.0
Attachments: HIVE-2822.03-branch0-8.patch, HIVE-2822.03.patch,
HIVE-2822.03b.patch, HIVE-2822.04-branch-08.patch,
HIVE-2822.05-branch0-8-1.patch, HIVE-2822.05-branch0-8.patch,
HIVE-2822.05.patch, HIVE-2822.D2475.1.patch, HIVE-2822.D2475.2.patch,
HIVE-2822.D2475.3.patch, HIVE-2822.D2475.4.patch,
hive-json-01-branch0-8.patch, hive-json-01.patch,
hive-json-02-branch0-8.patch, hive-json-02.patch
The goal is to have an option to produce JSON output of the DDL commands that
is easily machine parseable.
For example, desc my_table currently gives
{noformat}
idbigint
user string
{noformat}
and we want to allow a json output:
{noformat}
{
columns: [
{name: id, type: bigint},
{name: user, type: string}
]
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2559) Add target to install Hive JARs/POMs in the local Maven cache
[ https://issues.apache.org/jira/browse/HIVE-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2559: --- Resolution: Fixed Fix Version/s: 0.9.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Alan! Add target to install Hive JARs/POMs in the local Maven cache - Key: HIVE-2559 URL: https://issues.apache.org/jira/browse/HIVE-2559 Project: Hive Issue Type: Improvement Components: Build Infrastructure Affects Versions: 0.9.0 Reporter: Alejandro Abdelnur Assignee: Alan Gates Priority: Critical Fix For: 0.9.0 Attachments: HIVE-2559.patch HIVE-2391 is producing usable Maven artifacts. However, it only as a target to deploy/publish those artifacts to Apache Maven repos. There should be a new target to locally install Hive Maven artifacts, thus enabling their use from other projects before they are committed/publish to Apache Maven (this is critical to test patches that may address issues in downstream components). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2879) Add rat target to build to look for missing license headers
[ https://issues.apache.org/jira/browse/HIVE-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2879: --- Resolution: Fixed Fix Version/s: 0.9.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Owen! Add rat target to build to look for missing license headers - Key: HIVE-2879 URL: https://issues.apache.org/jira/browse/HIVE-2879 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.9.0 Attachments: HIVE-2879.D2373.1.patch, HIVE-2879.D2373.2.patch, HIVE-2879.D2373.3.patch We should have a rat target that can generate an Apache RAT report. RAT is used for finding missing license headers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247075#comment-13247075 ] Ashutosh Chauhan commented on HIVE-2764: Those tests passed. So, thats a progress. But following new ones failed. * rcfile_merge3.q * rcfile_createas1.q Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764.D2205.4.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2822) Add JSON output to the hive ddl commands
[
https://issues.apache.org/jira/browse/HIVE-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247079#comment-13247079
]
Hudson commented on HIVE-2822:
--
Integrated in Hive-trunk-h0.21 #1353 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/1353/])
HIVE-2822: Second part of it (ctdean via Ashutosh Chauhan) (Revision
1309667)
HIVE-2822 [jira] Add JSON output to the hive ddl commands
(Chris Dean via Ashutosh Chauhan)
Summary:
JSON output for DDL commands
By setting the variable hive.format=json then ddl commands will
produce json output suitable for client side parsing. For example, to
list all the databases one might get:
{
databases: [
default
]
}
The goal is to have an option to produce JSON output of the DDL commands that is
easily machine parseable.
For example, desc my_table currently gives
idbigint
user string
and we want to allow a json output:
{
columns: [
{name: id, type: bigint},
{name: user, type: string}
]
}
Test Plan: Run the provided unit tests
Reviewers: JIRA, ashutoshc
Reviewed By: ashutoshc
Differential Revision: https://reviews.facebook.net/D2475 (Revision 1309666)
Result = FAILURE
hashutosh :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309667
Files :
* /hive/trunk/ql/src/test/results/clientnegative/column_rename1.q.out
* /hive/trunk/ql/src/test/results/clientnegative/column_rename4.q.out
hashutosh :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309666
Files :
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml.template
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MapBuilder.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatter.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
* /hive/trunk/ql/src/test/queries/clientpositive/describe_database_json.q
* /hive/trunk/ql/src/test/queries/clientpositive/describe_table_json.q
* /hive/trunk/ql/src/test/queries/clientpositive/misc_json.q
* /hive/trunk/ql/src/test/queries/clientpositive/partitions_json.q
*
/hive/trunk/ql/src/test/results/clientnegative/database_create_already_exists.q.out
*
/hive/trunk/ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out
* /hive/trunk/ql/src/test/results/clientpositive/describe_database_json.q.out
* /hive/trunk/ql/src/test/results/clientpositive/describe_table_json.q.out
* /hive/trunk/ql/src/test/results/clientpositive/misc_json.q.out
* /hive/trunk/ql/src/test/results/clientpositive/partitions_json.q.out
Add JSON output to the hive ddl commands
Key: HIVE-2822
URL: https://issues.apache.org/jira/browse/HIVE-2822
Project: Hive
Issue Type: Improvement
Reporter: Chris Dean
Assignee: Chris Dean
Fix For: 0.9.0
Attachments: HIVE-2822.03-branch0-8.patch, HIVE-2822.03.patch,
HIVE-2822.03b.patch, HIVE-2822.04-branch-08.patch,
HIVE-2822.05-branch0-8-1.patch, HIVE-2822.05-branch0-8.patch,
HIVE-2822.05.patch, HIVE-2822.D2475.1.patch, HIVE-2822.D2475.2.patch,
HIVE-2822.D2475.3.patch, HIVE-2822.D2475.4.patch,
hive-json-01-branch0-8.patch, hive-json-01.patch,
hive-json-02-branch0-8.patch, hive-json-02.patch
The goal is to have an option to produce JSON output of the DDL commands that
is easily machine parseable.
For example, desc my_table currently gives
{noformat}
idbigint
user string
{noformat}
and we want to allow a json output:
{noformat}
{
columns: [
{name: id, type: bigint},
{name: user, type: string}
]
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2917) Add support for various charsets in LazySimpleSerDe
[ https://issues.apache.org/jira/browse/HIVE-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2917: -- Attachment: HIVE-2917.D2619.1.patch flyinggarden requested code review of HIVE-2917 [jira] Add support for various charsets in LazySimpleSerDe. Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2917 HIVE-2917: Add support for various charsets in LazySimpleSerDe Currently hive can only serialize/deserialize data encoded in utf-8. It would be useful to specify the data's charset when creating the table. The idea is to add a new keyword CHARSET to set charset at table level. For example: CREATE TABLE tbl1 (col1 STRING) ROW FORMAT CHARET GBK DELIMITED FIELDS TERMINATED BY '\t'; Another place to use CHARSET is in TRANSFORM clause. For example: SELECT TRANSFORM(col1, col2) ROW FORMAT CHARSET 'gbk' USING 'some_script' AS (col3, col4) ROW FORMAT CHARSET 'utf-8'; TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2619 AFFECTED FILES hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java data/files/gbk.txt serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyArrayMapStruct.java serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyCharset.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyString.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyUnionObjectInspector.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyStringObjectInspector.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyListObjectInspector.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyMapObjectInspector.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py serde/src/gen/thrift/gen-cpp/serde_constants.cpp serde/src/gen/thrift/gen-cpp/serde_constants.h serde/src/gen/thrift/gen-rb/serde_constants.rb serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java serde/src/gen/thrift/gen-php/serde/serde_constants.php serde/if/serde.thrift ql/src/test/results/clientpositive/charset.q.out ql/src/test/results/clientpositive/input35.q.out ql/src/test/results/clientpositive/input36.q.out ql/src/test/results/clientpositive/transform_charset.q.out ql/src/test/queries/clientpositive/transform_charset.q ql/src/test/queries/clientpositive/charset.q ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6027/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Add support for various charsets in LazySimpleSerDe --- Key: HIVE-2917 URL: https://issues.apache.org/jira/browse/HIVE-2917 Project: Hive Issue Type: New Feature Components: CLI, Serializers/Deserializers Affects Versions: 0.9.0 Reporter: Kai Zhang Attachments: HIVE-2917.1.patch.txt, HIVE-2917.2.patch.txt, HIVE-2917.3.patch.txt, HIVE-2917.D2619.1.patch Currently hive can only serialize/deserialize data encoded in utf-8. It would be useful to specify the data's charset when creating the table. The idea is to add a new keyword CHARSET to set charset at table level. For example: CREATE TABLE tbl1 (col1 STRING) ROW FORMAT CHARET GBK DELIMITED FIELDS TERMINATED BY '\t'; Another place to use CHARSET is in TRANSFORM clause. For example: SELECT TRANSFORM(col1, col2) ROW FORMAT CHARSET 'gbk' USING 'some_script' AS (col3, col4) ROW FORMAT CHARSET 'utf-8'; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA
[jira] [Assigned] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned HIVE-2923: -- Assignee: Thomas Weise (was: Ashutosh Chauhan) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac Key: HIVE-2923 URL: https://issues.apache.org/jira/browse/HIVE-2923 Project: Hive Issue Type: Bug Environment: Mac OSX Lion Reporter: Kevin Wilfong Assignee: Thomas Weise Priority: Blocker When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, it fails with the error: Failed to validate token path. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Failed to validate token path. at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) at org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) ... 17 more This message is also printed to standard out: Unable to load realm mapping info from SCDynamicStore The test seems to run fine in Linux, but more than one developer has reported this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2504) Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory
[ https://issues.apache.org/jira/browse/HIVE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated HIVE-2504: - Attachment: HIVE-2504.patch Made the newly created subdirectories get the same permissions as the parent by setting it explicitly on them. The table directories will get the permissions of the database directory and the partition directories will get the permissions of the table directory. Removed the hive.files.umask.value configuration parameter. Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory - Key: HIVE-2504 URL: https://issues.apache.org/jira/browse/HIVE-2504 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach Assignee: Rohini Palaniswamy Fix For: 0.9.0 Attachments: HIVE-2504.patch, HIVE-2504.patch When the Hive Metastore creates a subdirectory in the Hive warehouse for a new table it does so with the default HDFS permissions. Since the default dfs.umask value is 022, this means that the new subdirectory will not inherit the group write permissions of the hive warehouse directory. We should make the umask used by Warehouse.mkdirs() configurable, and set it to use a default value of 002. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2504) Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory
[ https://issues.apache.org/jira/browse/HIVE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated HIVE-2504: - Hadoop Flags: (was: Reviewed) Status: Patch Available (was: Reopened) Unit tests pass with ant clean package test Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory - Key: HIVE-2504 URL: https://issues.apache.org/jira/browse/HIVE-2504 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach Assignee: Rohini Palaniswamy Fix For: 0.9.0 Attachments: HIVE-2504.patch, HIVE-2504.patch When the Hive Metastore creates a subdirectory in the Hive warehouse for a new table it does so with the default HDFS permissions. Since the default dfs.umask value is 022, this means that the new subdirectory will not inherit the group write permissions of the hive warehouse directory. We should make the umask used by Warehouse.mkdirs() configurable, and set it to use a default value of 002. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Fix Version/s: 0.9.0 Affects Version/s: 0.9.0 Status: Patch Available (was: Open) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac Key: HIVE-2923 URL: https://issues.apache.org/jira/browse/HIVE-2923 Project: Hive Issue Type: Bug Affects Versions: 0.9.0 Environment: Mac OSX Lion Reporter: Kevin Wilfong Assignee: Thomas Weise Priority: Blocker Fix For: 0.9.0 Attachments: HIVE-2923.patch When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, it fails with the error: Failed to validate token path. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Failed to validate token path. at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) at org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) ... 17 more This message is also printed to standard out: Unable to load realm mapping info from SCDynamicStore The test seems to run fine in Linux, but more than one developer has reported this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Attachment: HIVE-2923.patch The problem is that the ZooKeeper client is used before it is connected (may or may not occur depending on environment). I could reproduce this yesterday on my Mac but not this morning. Attached patch will add a wait for SyncConnected with timeout to address this. Can you please try the patch? testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac Key: HIVE-2923 URL: https://issues.apache.org/jira/browse/HIVE-2923 Project: Hive Issue Type: Bug Affects Versions: 0.9.0 Environment: Mac OSX Lion Reporter: Kevin Wilfong Assignee: Thomas Weise Priority: Blocker Fix For: 0.9.0 Attachments: HIVE-2923.patch When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, it fails with the error: Failed to validate token path. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Failed to validate token path. at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) at org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) ... 17 more This message is also printed to standard out: Unable to load realm mapping info from SCDynamicStore The test seems to run fine in Linux, but more than one developer has reported this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2927) Allow escape character in get_json_object
Allow escape character in get_json_object
-
Key: HIVE-2927
URL: https://issues.apache.org/jira/browse/HIVE-2927
Project: Hive
Issue Type: Improvement
Components: Serializers/Deserializers
Affects Versions: 0.8.1
Reporter: Sean McNamara
Fix For: 0.9.0
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like so:
{{get_json_object(json, '$.c\\.d') would return: 2}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[
https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean McNamara updated HIVE-2927:
Component/s: (was: Serializers/Deserializers)
Security
Description:
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like so:
{{get_json_object(json, '$.c.d') would return: 2}}
was:
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like so:
{{get_json_object(json, '$.c\\.d') would return: 2}}
Affects Version/s: (was: 0.8.1)
Fix Version/s: (was: 0.9.0)
Allow escape character in get_json_object
-
Key: HIVE-2927
URL: https://issues.apache.org/jira/browse/HIVE-2927
Project: Hive
Issue Type: Improvement
Components: Security
Reporter: Sean McNamara
Original Estimate: 0h
Remaining Estimate: 0h
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like
so:
{{get_json_object(json, '$.c.d') would return: 2}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[
https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean McNamara updated HIVE-2927:
Description:
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like so:
{{get_json_object(json, '$.c\\\.d') would return: 2}}
was:
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like so:
{{get_json_object(json, '$.c.d') would return: 2}}
Allow escape character in get_json_object
-
Key: HIVE-2927
URL: https://issues.apache.org/jira/browse/HIVE-2927
Project: Hive
Issue Type: Improvement
Components: Security
Reporter: Sean McNamara
Original Estimate: 0h
Remaining Estimate: 0h
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like
so:
{{get_json_object(json, '$.c\\\.d') would return: 2}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[
https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean McNamara updated HIVE-2927:
Attachment: HIVE-2927.1.patch.txt
Patch adds ability to escape '.' in JSON keys using \\.
Allow escape character in get_json_object
-
Key: HIVE-2927
URL: https://issues.apache.org/jira/browse/HIVE-2927
Project: Hive
Issue Type: Improvement
Components: Security
Reporter: Sean McNamara
Attachments: HIVE-2927.1.patch.txt
Original Estimate: 0h
Remaining Estimate: 0h
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like
so:
{{get_json_object(json, '$.c\\\.d') would return: 2}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[
https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean McNamara updated HIVE-2927:
Needs code review. thnx!
Allow escape character in get_json_object
-
Key: HIVE-2927
URL: https://issues.apache.org/jira/browse/HIVE-2927
Project: Hive
Issue Type: Improvement
Components: Security
Reporter: Sean McNamara
Attachments: HIVE-2927.1.patch.txt
Original Estimate: 0h
Remaining Estimate: 0h
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like
so:
{{get_json_object(json, '$.c\\\.d') would return: 2}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[
https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean McNamara updated HIVE-2927:
Component/s: (was: Security)
Serializers/Deserializers
Allow escape character in get_json_object
-
Key: HIVE-2927
URL: https://issues.apache.org/jira/browse/HIVE-2927
Project: Hive
Issue Type: Improvement
Components: Serializers/Deserializers
Reporter: Sean McNamara
Attachments: HIVE-2927.1.patch.txt
Original Estimate: 0h
Remaining Estimate: 0h
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like
so:
{{get_json_object(json, '$.c\\\.d') would return: 2}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2711) Make the header of RCFile unique
[ https://issues.apache.org/jira/browse/HIVE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247463#comment-13247463 ] Phabricator commented on HIVE-2711: --- ashutoshc has accepted the revision HIVE-2711 [jira] Make the header of RCFile unique. +1 REVISION DETAIL https://reviews.facebook.net/D2115 BRANCH h-2711 Make the header of RCFile unique Key: HIVE-2711 URL: https://issues.apache.org/jira/browse/HIVE-2711 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-2711.D2115.1.patch, HIVE-2711.D2115.2.patch, HIVE-2711.D2115.3.patch, HIVE-2711.D2571.1.patch The RCFile implementation was copied from Hadoop's SequenceFile and copied the 'magic' string in the header. This means that you can't use the header to distinguish between RCFiles and SequenceFiles. I'd propose that we create a new header for RCFiles (RCF?) to replace the current SEQ. To maintain compatibility, we'll need to continue to accept the current 'SEQ\06' and just make new files contain the new header. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2711) Make the header of RCFile unique
[ https://issues.apache.org/jira/browse/HIVE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-2711: Attachment: rc-file-v0.rc Here's the binary file that arc didn't include in the patch. Make the header of RCFile unique Key: HIVE-2711 URL: https://issues.apache.org/jira/browse/HIVE-2711 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-2711.D2115.1.patch, HIVE-2711.D2115.2.patch, HIVE-2711.D2115.3.patch, HIVE-2711.D2571.1.patch, rc-file-v0.rc The RCFile implementation was copied from Hadoop's SequenceFile and copied the 'magic' string in the header. This means that you can't use the header to distinguish between RCFiles and SequenceFiles. I'd propose that we create a new header for RCFiles (RCF?) to replace the current SEQ. To maintain compatibility, we'll need to continue to accept the current 'SEQ\06' and just make new files contain the new header. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1354 - Failure
Changes for Build #1354
[hashutosh] HIVE-2879 : Add rat target to build to look for missing license
headers (Owen Omalley via Ashutosh Chauhan)
[hashutosh] HIVE-2559 : Add target to install Hive JARs/POMs in the local Maven
cache (Alan Gates via Ashutosh Chauhan)
[hashutosh] HIVE-2822: Second part of it (ctdean via Ashutosh Chauhan)
[hashutosh] HIVE-2822 [jira] Add JSON output to the hive ddl commands
(Chris Dean via Ashutosh Chauhan)
Summary:
JSON output for DDL commands
By setting the variable hive.format=json then ddl commands will
produce json output suitable for client side parsing. For example, to
list all the databases one might get:
{
databases: [
default
]
}
The goal is to have an option to produce JSON output of the DDL commands that is
easily machine parseable.
For example, desc my_table currently gives
idbigint
user string
and we want to allow a json output:
{
columns: [
{name: id, type: bigint},
{name: user, type: string}
]
}
Test Plan: Run the provided unit tests
Reviewers: JIRA, ashutoshc
Reviewed By: ashutoshc
Differential Revision: https://reviews.facebook.net/D2475
1 tests failed.
REGRESSION:
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1
Error Message:
Unexpected exception See build/ql/tmp/hive.log, or try ant test ...
-Dtest.silent=false to get more logs.
Stack Trace:
junit.framework.AssertionFailedError: Unexpected exception
See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get
more logs.
at junit.framework.Assert.fail(Assert.java:47)
at
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:10418)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785)
The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1354)
Status: Failure
Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1354/ to
view the results.
[jira] [Commented] (HIVE-2879) Add rat target to build to look for missing license headers
[ https://issues.apache.org/jira/browse/HIVE-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247536#comment-13247536 ] Hudson commented on HIVE-2879: -- Integrated in Hive-trunk-h0.21 #1354 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1354/]) HIVE-2879 : Add rat target to build to look for missing license headers (Owen Omalley via Ashutosh Chauhan) (Revision 1309676) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309676 Files : * /hive/trunk/build.xml * /hive/trunk/ivy.xml * /hive/trunk/ivy/libraries.properties Add rat target to build to look for missing license headers - Key: HIVE-2879 URL: https://issues.apache.org/jira/browse/HIVE-2879 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.9.0 Attachments: HIVE-2879.D2373.1.patch, HIVE-2879.D2373.2.patch, HIVE-2879.D2373.3.patch We should have a rat target that can generate an Apache RAT report. RAT is used for finding missing license headers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2559) Add target to install Hive JARs/POMs in the local Maven cache
[ https://issues.apache.org/jira/browse/HIVE-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247535#comment-13247535 ] Hudson commented on HIVE-2559: -- Integrated in Hive-trunk-h0.21 #1354 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1354/]) HIVE-2559 : Add target to install Hive JARs/POMs in the local Maven cache (Alan Gates via Ashutosh Chauhan) (Revision 1309675) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309675 Files : * /hive/trunk/build.xml Add target to install Hive JARs/POMs in the local Maven cache - Key: HIVE-2559 URL: https://issues.apache.org/jira/browse/HIVE-2559 Project: Hive Issue Type: Improvement Components: Build Infrastructure Affects Versions: 0.9.0 Reporter: Alejandro Abdelnur Assignee: Alan Gates Priority: Critical Fix For: 0.9.0 Attachments: HIVE-2559.patch HIVE-2391 is producing usable Maven artifacts. However, it only as a target to deploy/publish those artifacts to Apache Maven repos. There should be a new target to locally install Hive Maven artifacts, thus enabling their use from other projects before they are committed/publish to Apache Maven (this is critical to test patches that may address issues in downstream components). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2822) Add JSON output to the hive ddl commands
[
https://issues.apache.org/jira/browse/HIVE-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247537#comment-13247537
]
Hudson commented on HIVE-2822:
--
Integrated in Hive-trunk-h0.21 #1354 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/1354/])
HIVE-2822: Second part of it (ctdean via Ashutosh Chauhan) (Revision
1309667)
HIVE-2822 [jira] Add JSON output to the hive ddl commands
(Chris Dean via Ashutosh Chauhan)
Summary:
JSON output for DDL commands
By setting the variable hive.format=json then ddl commands will
produce json output suitable for client side parsing. For example, to
list all the databases one might get:
{
databases: [
default
]
}
The goal is to have an option to produce JSON output of the DDL commands that is
easily machine parseable.
For example, desc my_table currently gives
idbigint
user string
and we want to allow a json output:
{
columns: [
{name: id, type: bigint},
{name: user, type: string}
]
}
Test Plan: Run the provided unit tests
Reviewers: JIRA, ashutoshc
Reviewed By: ashutoshc
Differential Revision: https://reviews.facebook.net/D2475 (Revision 1309666)
Result = FAILURE
hashutosh :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309667
Files :
* /hive/trunk/ql/src/test/results/clientnegative/column_rename1.q.out
* /hive/trunk/ql/src/test/results/clientnegative/column_rename4.q.out
hashutosh :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309666
Files :
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml.template
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MapBuilder.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatter.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
* /hive/trunk/ql/src/test/queries/clientpositive/describe_database_json.q
* /hive/trunk/ql/src/test/queries/clientpositive/describe_table_json.q
* /hive/trunk/ql/src/test/queries/clientpositive/misc_json.q
* /hive/trunk/ql/src/test/queries/clientpositive/partitions_json.q
*
/hive/trunk/ql/src/test/results/clientnegative/database_create_already_exists.q.out
*
/hive/trunk/ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out
* /hive/trunk/ql/src/test/results/clientpositive/describe_database_json.q.out
* /hive/trunk/ql/src/test/results/clientpositive/describe_table_json.q.out
* /hive/trunk/ql/src/test/results/clientpositive/misc_json.q.out
* /hive/trunk/ql/src/test/results/clientpositive/partitions_json.q.out
Add JSON output to the hive ddl commands
Key: HIVE-2822
URL: https://issues.apache.org/jira/browse/HIVE-2822
Project: Hive
Issue Type: Improvement
Reporter: Chris Dean
Assignee: Chris Dean
Fix For: 0.9.0
Attachments: HIVE-2822.03-branch0-8.patch, HIVE-2822.03.patch,
HIVE-2822.03b.patch, HIVE-2822.04-branch-08.patch,
HIVE-2822.05-branch0-8-1.patch, HIVE-2822.05-branch0-8.patch,
HIVE-2822.05.patch, HIVE-2822.D2475.1.patch, HIVE-2822.D2475.2.patch,
HIVE-2822.D2475.3.patch, HIVE-2822.D2475.4.patch,
hive-json-01-branch0-8.patch, hive-json-01.patch,
hive-json-02-branch0-8.patch, hive-json-02.patch
The goal is to have an option to produce JSON output of the DDL commands that
is easily machine parseable.
For example, desc my_table currently gives
{noformat}
idbigint
user string
{noformat}
and we want to allow a json output:
{noformat}
{
columns: [
{name: id, type: bigint},
{name: user, type: string}
]
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[
https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean McNamara updated HIVE-2927:
Description:
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, $.a.b) returns: 1}}
{{get_json_object(json, $.c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like so:
{{get_json_object(json, '$.c\\\.d') would return: 2}}
was:
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, a.b) returns: 1}}
{{get_json_object(json, c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like so:
{{get_json_object(json, '$.c\\\.d') would return: 2}}
Affects Version/s: 0.8.1
Fix Version/s: 0.9.0
Allow escape character in get_json_object
-
Key: HIVE-2927
URL: https://issues.apache.org/jira/browse/HIVE-2927
Project: Hive
Issue Type: Improvement
Components: Serializers/Deserializers
Affects Versions: 0.8.1
Reporter: Sean McNamara
Fix For: 0.9.0
Attachments: HIVE-2927.1.patch.txt
Original Estimate: 0h
Remaining Estimate: 0h
*Background:*
get_json_object extracts json objects from a json string based on a specified
path.
*Problem:*
The current implementation of get_json_object can't see keys with a '.' in
them. Our data contains '.' in the keys, so we have to filter our json keys
through a streaming script to replace '.' for '_'.
*Example:*
{{json = {a:{b: 1}, c.d: 2}}}
{{get_json_object(json, $.a.b) returns: 1}}
{{get_json_object(json, $.c.d) returns: NULL}}
In the present implementation of get_json_object, c.d is not addressable.
*Proposal:*
The desired behavior would be to allow the JSON path to be escape-able, like
so:
{{get_json_object(json, '$.c\\\.d') would return: 2}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-0.8.1-SNAPSHOT-h0.21 - Build # 244 - Failure
Changes for Build #244 1 tests failed. REGRESSION: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:9440) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) The Apache Jenkins build system has built Hive-0.8.1-SNAPSHOT-h0.21 (build #244) Status: Failure Check console output at https://builds.apache.org/job/Hive-0.8.1-SNAPSHOT-h0.21/244/ to view the results.
[jira] [Commented] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247679#comment-13247679 ] Phabricator commented on HIVE-2909: --- njain has commented on the revision HIVE-2909 [jira] SHOW COLUMNS table_name; to provide a comma-delimited list of columns.. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:2037 dont sort the columns. Use the same order as used by describeTable REVISION DETAIL https://reviews.facebook.net/D2595 SHOW COLUMNS table_name; to provide a comma-delimited list of columns. -- Key: HIVE-2909 URL: https://issues.apache.org/jira/browse/HIVE-2909 Project: Hive Issue Type: New Feature Reporter: Adam Kramer Assignee: Dikang Gu Priority: Minor Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, HIVE-2909.D2595.3.patch Due to the way that SELECT * and partitioning works, it is frequently obnoxious to insert data into tables of the same schema. This could be fixed in a number of ways, all murky; this feature request reduces the obnoxicity of the current situation. SHOW COLUMNS foo; OK bar, baz, tball, ds ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2909: -- Attachment: HIVE-2909.D2595.4.patch dikanggu updated the revision HIVE-2909 [jira] SHOW COLUMNS table_name; to provide a comma-delimited list of columns.. Reviewers: JIRA, njain address Namit's comments. REVISION DETAIL https://reviews.facebook.net/D2595 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java ql/src/java/org/apache/hadoop/hive/ql/plan/ShowColumnsDesc.java ql/src/test/queries/clientpositive/show_columns.q ql/src/test/results/clientpositive/show_columns.q.out SHOW COLUMNS table_name; to provide a comma-delimited list of columns. -- Key: HIVE-2909 URL: https://issues.apache.org/jira/browse/HIVE-2909 Project: Hive Issue Type: New Feature Reporter: Adam Kramer Assignee: Dikang Gu Priority: Minor Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch Due to the way that SELECT * and partitioning works, it is frequently obnoxious to insert data into tables of the same schema. This could be fixed in a number of ways, all murky; this feature request reduces the obnoxicity of the current situation. SHOW COLUMNS foo; OK bar, baz, tball, ds ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247743#comment-13247743 ] Ashutosh Chauhan commented on HIVE-2926: Couple of comments: * This assumes sh, cat, sed etc. to be available in the environment. So, it may not work where these tools are not present, which is fine but thought to point this out. I tested on mac and linux, it builds fine. * I don't see how your generated package-info.java will get compiled. You create a .java file in build dir. Have you tested that it works as you expect? Expose some information about the metastore through JMX --- Key: HIVE-2926 URL: https://issues.apache.org/jira/browse/HIVE-2926 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.9.0 Attachments: metastore-jmx.patch Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247759#comment-13247759 ] Carl Steinbach commented on HIVE-2926: -- bin/hive uses Bash. I think we should do the same thing here. Also note that on most platforms /bin/sh actually maps to bash, or to something entirely different (e.g. https://wiki.ubuntu.com/DashAsBinSh). Expose some information about the metastore through JMX --- Key: HIVE-2926 URL: https://issues.apache.org/jira/browse/HIVE-2926 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.9.0 Attachments: metastore-jmx.patch Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247771#comment-13247771 ] Phabricator commented on HIVE-2909: --- njain has commented on the revision HIVE-2909 [jira] SHOW COLUMNS table_name; to provide a comma-delimited list of columns.. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:2039 for partitioned tables, all partitioned columns also cols.addAll(tbl.getPartCols()); REVISION DETAIL https://reviews.facebook.net/D2595 SHOW COLUMNS table_name; to provide a comma-delimited list of columns. -- Key: HIVE-2909 URL: https://issues.apache.org/jira/browse/HIVE-2909 Project: Hive Issue Type: New Feature Reporter: Adam Kramer Assignee: Dikang Gu Priority: Minor Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch Due to the way that SELECT * and partitioning works, it is frequently obnoxious to insert data into tables of the same schema. This could be fixed in a number of ways, all murky; this feature request reduces the obnoxicity of the current situation. SHOW COLUMNS foo; OK bar, baz, tball, ds ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-2926: -- Attachment: hive-2926.1.patch @Ashutosh the patch attached here has been tested. Yes this assumes executables like 'sed'. I don't have a better solution at this point for handling that issue. The way version is derived in this patch is similar to the way Hadoop/HBase does.. @Carl, I changed the 'sh' to 'bash' Expose some information about the metastore through JMX --- Key: HIVE-2926 URL: https://issues.apache.org/jira/browse/HIVE-2926 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.9.0 Attachments: hive-2926.1.patch, metastore-jmx.patch Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247784#comment-13247784 ] Carl Steinbach commented on HIVE-2926: -- Sounds good. Please submit a phabricator request (https://cwiki.apache.org/Hive/phabricatorcodereview.html) when the patch is ready for review. Expose some information about the metastore through JMX --- Key: HIVE-2926 URL: https://issues.apache.org/jira/browse/HIVE-2926 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.9.0 Attachments: hive-2926.1.patch, metastore-jmx.patch Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2863) Ambiguous table name or column reference message displays when table and column names are the same
[ https://issues.apache.org/jira/browse/HIVE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2863: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Navis Ambiguous table name or column reference message displays when table and column names are the same -- Key: HIVE-2863 URL: https://issues.apache.org/jira/browse/HIVE-2863 Project: Hive Issue Type: Bug Reporter: Mauro Cazzari Assignee: Navis Attachments: HIVE-2863.D2361.1.patch Given the following table: CREATE TABLE `Y` (`y` DOUBLE) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS TEXTFILE; The following query fails: SELECT `Y`.`y` FROM `Y` WHERE ( `y` = 1 ) ERROR: java.sql.SQLException: Query returned non-zero code: 10, cause: FAILED: Error in semantic analysis: Line 1:36 Ambiguous table alias or column reference '`y`' ERROR: Unable to execute Hadoop query. ERROR: Prepare error. SQL statement: SELECT `Y`.`y` FROM `Y` WHERE ( `y` = 1 ). The problem goes away if the table and column names do not match. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Attachment: (was: HIVE-2923.patch) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac Key: HIVE-2923 URL: https://issues.apache.org/jira/browse/HIVE-2923 Project: Hive Issue Type: Bug Affects Versions: 0.9.0 Environment: Mac OSX Lion Reporter: Kevin Wilfong Assignee: Thomas Weise Priority: Blocker Fix For: 0.9.0 Attachments: HIVE-2923.patch When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, it fails with the error: Failed to validate token path. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Failed to validate token path. at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) at org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) ... 17 more This message is also printed to standard out: Unable to load realm mapping info from SCDynamicStore The test seems to run fine in Linux, but more than one developer has reported this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2926: -- Attachment: HIVE-2926.D2637.1.patch ddraj requested code review of HIVE-2926 [jira] Expose some information about the metastore through JMX. Reviewers: JIRA Adds VersionInfo and makes some metastore information exposed through JMX Expose some information about the metastore through JMX TEST PLAN Did manual testing of the JMX stuff (including the version). Accessed the data through jconsole. REVISION DETAIL https://reviews.facebook.net/D2637 AFFECTED FILES common/build.xml common/saveVersion.sh common/src/java/org/apache/hadoop/hive/common/HiveVersionAnnotation.java common/src/java/org/apache/hadoop/hive/common/VersionInfo.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/MXBean.java metastore/src/java/org/apache/hadoop/hive/metastore/MXBeanImpl.java ql/src/test/results/clientnegative/column_rename1.q.out ql/src/test/results/clientnegative/column_rename4.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6063/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Expose some information about the metastore through JMX --- Key: HIVE-2926 URL: https://issues.apache.org/jira/browse/HIVE-2926 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.9.0 Attachments: HIVE-2926.D2637.1.patch, hive-2926.1.patch, metastore-jmx.patch Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Attachment: HIVE-2923.patch Updated patch to increase timeout for slow test environments. testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac Key: HIVE-2923 URL: https://issues.apache.org/jira/browse/HIVE-2923 Project: Hive Issue Type: Bug Affects Versions: 0.9.0 Environment: Mac OSX Lion Reporter: Kevin Wilfong Assignee: Thomas Weise Priority: Blocker Fix For: 0.9.0 Attachments: HIVE-2923.patch When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, it fails with the error: Failed to validate token path. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Failed to validate token path. at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) at org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) ... 17 more This message is also printed to standard out: Unable to load realm mapping info from SCDynamicStore The test seems to run fine in Linux, but more than one developer has reported this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247820#comment-13247820 ] Phabricator commented on HIVE-2926: --- ddraj has added reviewers to the revision HIVE-2926 [jira] Expose some information about the metastore through JMX. Added Reviewers: ashutoshc, khorgath, cwsteinbach REVISION DETAIL https://reviews.facebook.net/D2637 Expose some information about the metastore through JMX --- Key: HIVE-2926 URL: https://issues.apache.org/jira/browse/HIVE-2926 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.9.0 Attachments: HIVE-2926.D2637.1.patch, hive-2926.1.patch, metastore-jmx.patch Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2901) Hive union with NULL constant and string in same column returns all null
[ https://issues.apache.org/jira/browse/HIVE-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247829#comment-13247829 ] Phabricator commented on HIVE-2901: --- njain has accepted the revision HIVE-2901 [jira] Hive union with NULL constant and string in same column returns all null. REVISION DETAIL https://reviews.facebook.net/D2541 BRANCH DPAL-1029 Hive union with NULL constant and string in same column returns all null Key: HIVE-2901 URL: https://issues.apache.org/jira/browse/HIVE-2901 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong Assignee: Navis Priority: Critical Attachments: HIVE-2901.D2541.1.patch, HIVE-2901.D2541.2.patch, HIVE-2901.D2541.3.patch select x from (select value as x from src union all select NULL as x from src)a; This query produces all nulls, where value is a string column. Notably, select x from (select key as x from src union all select NULL as x from src)a; where key is a string, but can be cast to a double, the query returns correct results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247840#comment-13247840 ] Phabricator commented on HIVE-2926: --- ddraj has commented on the revision HIVE-2926 [jira] Expose some information about the metastore through JMX. Ignore the two patch changes in *.out files. I don't know how these crept in. REVISION DETAIL https://reviews.facebook.net/D2637 Expose some information about the metastore through JMX --- Key: HIVE-2926 URL: https://issues.apache.org/jira/browse/HIVE-2926 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.9.0 Attachments: HIVE-2926.D2637.1.patch, hive-2926.1.patch, metastore-jmx.patch Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247852#comment-13247852 ] Phabricator commented on HIVE-2926: --- cwsteinbach has requested changes to the revision HIVE-2926 [jira] Expose some information about the metastore through JMX. INLINE COMMENTS common/src/java/org/apache/hadoop/hive/common/HiveVersionAnnotation.java:26 Comment should reference Hive instead of Hadoop. Same thing applies to several of the comments below. common/src/java/org/apache/hadoop/hive/common/VersionInfo.java:101 Change to Apache Hive ql/src/test/results/clientnegative/column_rename1.q.out:26 These q file updates don't belong here, right? common/build.xml:30 I think this target should be moved into build-common.xml and should get called for all sub-projects. This will make it a lot easier to diagnose cases where people are using JARs from different versions of Hive. common/src/java/org/apache/hadoop/hive/common/HiveVersionAnnotation.java:50 According to saveVersion.sh this could also return the URL for a Git repo. The comment should be updated to reflect this, and I think the name of the method should be changed to getRepositoryUrl() common/src/java/org/apache/hadoop/hive/common/HiveVersionAnnotation.java:55 Ditto. common/src/java/org/apache/hadoop/hive/common/VersionInfo.java:75 source code repository URL common/src/java/org/apache/hadoop/hive/common/VersionInfo.java:95 Does it make sense to reference a revision number without also mentioning the repository URL? common/src/java/org/apache/hadoop/hive/common/VersionInfo.java:102 This references svn. What if this was built using git? metastore/src/java/org/apache/hadoop/hive/metastore/MXBeanImpl.java:33 This assumes that VersionInfo.getRevision() returns an SVN revision number. metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:3060 Does it make sense to include minWorkerThreads, or the value of hive.metastore.uris? What about the value of useSasl or tcpKeepAlive? REVISION DETAIL https://reviews.facebook.net/D2637 BRANCH HIVE-2926-branch Expose some information about the metastore through JMX --- Key: HIVE-2926 URL: https://issues.apache.org/jira/browse/HIVE-2926 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.9.0 Attachments: HIVE-2926.D2637.1.patch, hive-2926.1.patch, metastore-jmx.patch Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247862#comment-13247862 ] Ashutosh Chauhan commented on HIVE-2923: Thanks, Thomas for taking this up. Very much appreciated! testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac Key: HIVE-2923 URL: https://issues.apache.org/jira/browse/HIVE-2923 Project: Hive Issue Type: Bug Affects Versions: 0.9.0 Environment: Mac OSX Lion Reporter: Kevin Wilfong Assignee: Thomas Weise Priority: Blocker Fix For: 0.9.0 Attachments: HIVE-2923.patch When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, it fails with the error: Failed to validate token path. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Failed to validate token path. at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) at org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) ... 17 more This message is also printed to standard out: Unable to load realm mapping info from SCDynamicStore The test seems to run fine in Linux, but more than one developer has reported this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247860#comment-13247860 ] Ashutosh Chauhan commented on HIVE-2923: I was able to reproduce failures before the patch. With the latest patch, failures went away and tests pass. Patch looks good to me. Those who were seeing failures earlier can try with this patch and report back. testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac Key: HIVE-2923 URL: https://issues.apache.org/jira/browse/HIVE-2923 Project: Hive Issue Type: Bug Affects Versions: 0.9.0 Environment: Mac OSX Lion Reporter: Kevin Wilfong Assignee: Thomas Weise Priority: Blocker Fix For: 0.9.0 Attachments: HIVE-2923.patch When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, it fails with the error: Failed to validate token path. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Failed to validate token path. at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) at org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) ... 17 more This message is also printed to standard out: Unable to load realm mapping info from SCDynamicStore The test seems to run fine in Linux, but more than one developer has reported this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2585) Collapse hive.metastore.uris and hive.metastore.local
[ https://issues.apache.org/jira/browse/HIVE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247875#comment-13247875 ] Phabricator commented on HIVE-2585: --- cwsteinbach has commented on the revision HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local. INLINE COMMENTS conf/hive-default.xml.template:102 HiveConf.init() seems like the right place to make these checks. REVISION DETAIL https://reviews.facebook.net/D2559 BRANCH svn Collapse hive.metastore.uris and hive.metastore.local - Key: HIVE-2585 URL: https://issues.apache.org/jira/browse/HIVE-2585 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-2585.D2559.1.patch We should just have hive.metastore.uris. If it is empty, we shall assume local mode, if non-empty we shall use that string to connect to remote metastore. Having two different keys for same information is confusing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2585) Collapse hive.metastore.uris and hive.metastore.local
[ https://issues.apache.org/jira/browse/HIVE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247885#comment-13247885 ] Phabricator commented on HIVE-2585: --- thw has commented on the revision HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local. INLINE COMMENTS conf/hive-default.xml.template:102 A scenario where this would lead to change of existing behavior is when client and metastore server point to the same conf, with client using the thrift URI and server javax.jdo.option.ConnectionURL (should only happen in dev environments). REVISION DETAIL https://reviews.facebook.net/D2559 BRANCH svn Collapse hive.metastore.uris and hive.metastore.local - Key: HIVE-2585 URL: https://issues.apache.org/jira/browse/HIVE-2585 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-2585.D2559.1.patch We should just have hive.metastore.uris. If it is empty, we shall assume local mode, if non-empty we shall use that string to connect to remote metastore. Having two different keys for same information is confusing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247891#comment-13247891 ] Carl Steinbach commented on HIVE-2909: -- MySQL has a SHOW COLUMNS command: http://dev.mysql.com/doc/refman/5.0/en/show-columns.html Seems like we should follow convention and try to duplicate that instead of inventing our own behavior. Also, I think returning each column name in a separate row is much more convenient (and consistent) for clients than forcing them to tokenize the names by splitting on ',' SHOW COLUMNS table_name; to provide a comma-delimited list of columns. -- Key: HIVE-2909 URL: https://issues.apache.org/jira/browse/HIVE-2909 Project: Hive Issue Type: New Feature Reporter: Adam Kramer Assignee: Dikang Gu Priority: Minor Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch Due to the way that SELECT * and partitioning works, it is frequently obnoxious to insert data into tables of the same schema. This could be fixed in a number of ways, all murky; this feature request reduces the obnoxicity of the current situation. SHOW COLUMNS foo; OK bar, baz, tball, ds ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.8.1 Reporter: Mithun Radhakrishnan I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: quote Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 /quote But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-2928: --- Affects Version/s: (was: 0.8.1) 0.9.0 Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.9.0 Reporter: Mithun Radhakrishnan Attachments: HIVE-2928.patch I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: quote Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 /quote But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2928) Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-2928: --- Attachment: HIVE-2928.patch Schema-creation in Oracle. + Changes to package.jdo (longvarchar to clob). Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.9.0 Reporter: Mithun Radhakrishnan Attachments: HIVE-2928.patch I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: quote Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 /quote But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2909: -- Attachment: HIVE-2909.D2595.5.patch dikanggu updated the revision HIVE-2909 [jira] SHOW COLUMNS table_name; to provide a comma-delimited list of columns.. Reviewers: JIRA, njain address Namit's comments, add the partition columns. REVISION DETAIL https://reviews.facebook.net/D2595 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java ql/src/java/org/apache/hadoop/hive/ql/plan/ShowColumnsDesc.java ql/src/test/queries/clientpositive/show_columns.q ql/src/test/results/clientpositive/show_columns.q.out SHOW COLUMNS table_name; to provide a comma-delimited list of columns. -- Key: HIVE-2909 URL: https://issues.apache.org/jira/browse/HIVE-2909 Project: Hive Issue Type: New Feature Reporter: Adam Kramer Assignee: Dikang Gu Priority: Minor Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch, HIVE-2909.D2595.5.patch Due to the way that SELECT * and partitioning works, it is frequently obnoxious to insert data into tables of the same schema. This could be fixed in a number of ways, all murky; this feature request reduces the obnoxicity of the current situation. SHOW COLUMNS foo; OK bar, baz, tball, ds ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-600) Running TPC-H queries on Hive
[ https://issues.apache.org/jira/browse/HIVE-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247949#comment-13247949 ] alex gemini commented on HIVE-600: -- I guess the Q9 failed is because hive set mapred.min.split.size=536870912, the default jvm setting for task launch is 200M, so it failed,remove this setting or set jvm memory for a larger value would be nice. Running TPC-H queries on Hive - Key: HIVE-600 URL: https://issues.apache.org/jira/browse/HIVE-600 Project: Hive Issue Type: New Feature Reporter: Yuntao Jia Assignee: Yuntao Jia Attachments: TPC-H_on_Hive_2009-08-11.pdf, TPC-H_on_Hive_2009-08-11.tar.gz, TPC-H_on_Hive_2009-08-14.tar.gz The goal is to run all TPC-H (http://www.tpc.org/tpch/) benchmark queries on Hive for two reasons. First, through those queries, we would like to find the new features that we need to put into Hive so that Hive supports common SQL queries. Second, we would like to measure the performance of Hive to find out what Hive is not good at. We can then improve Hive based on those information. For queries that are not supported now in Hive, I will try to rewrite them to one or more Hive-supported queries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1721) use bloom filters to improve the performance of joins
[ https://issues.apache.org/jira/browse/HIVE-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247985#comment-13247985 ] alex gemini commented on HIVE-1721: --- I'm wondering how we apply bloom filter to big table.we use map side join for small table 25M, if we use bloom filter build small table,we maybe can increase small table size to 200M, but in big table map stage,we need to read bloom filter and writer intermediate result back to disk and then reading this intermediate result to check the real small table,we still can't hold the actual real small table into memory(correct the logic if I'm wrong),we pay the cost of writer a intermediate result which is very close to final result.In this case we can't increase the map number because it will double the penalty of io.I guess it will only get benefit in three table join on same join key,one small with 2 big.In my opinion the other db system can get benefit of bloom filter is because they can hold the intermediate result in memory for further processing (like oracle) or print it immediate (like hbase). use bloom filters to improve the performance of joins - Key: HIVE-1721 URL: https://issues.apache.org/jira/browse/HIVE-1721 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Labels: gsoc, gsoc2012, optimization In case of map-joins, it is likely that the big table will not find many matching rows from the small table. Currently, we perform a hash-map lookup for every row in the big table, which can be pretty expensive. It might be useful to try out a bloom-filter containing all the elements in the small table. Each element from the big table is first searched in the bloom filter, and only in case of a positive match, the small table hash table is explored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Debugging hive tasks?
you can try to set mapred.job.reuse.jvm.num.tasks to higher value in mapred-site.xml conf file, this will enable jvm reuse. 在 2012年4月2日 上午2:56，Travis Crawford traviscrawf...@gmail.com写道： Hey hive gurus - On the wiki I found a section about running Hive jobs in local mode, however, it uses a separate JVM. Is it possible to run the map reduce jobs inside the current process? I'm able to attach a debugger to the CLI process but haven't been able to figure out debugging the MR tasks. Any tips would be appreciated (and I'll update the wiki). https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-Hive%2CMapReduceandLocalMode Thanks! Travis
