[jira] [Commented] (HIVE-1721) use bloom filters to improve the performance of joins
[ https://issues.apache.org/jira/browse/HIVE-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248115#comment-13248115 ] alex gemini commented on HIVE-1721: --- The original thought is increase the map side small table size,this is very depend on how we chunk big table.if big table is chunk into buckets for 16 buckets,the small table must be automatic change to 16 buckets too(same logic bucket map join).if big table is partition by (region string),the small table also need to partition by (region) first and also make sure the smallest chunk size is not bigger than current small table size define by hive.The partition case is more often,we can avoid common join by always let small table chunk like big table's format. > use bloom filters to improve the performance of joins > - > > Key: HIVE-1721 > URL: https://issues.apache.org/jira/browse/HIVE-1721 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain > Labels: gsoc, gsoc2012, optimization > > In case of map-joins, it is likely that the big table will not find many > matching rows from the small table. > Currently, we perform a hash-map lookup for every row in the big table, which > can be pretty expensive. > It might be useful to try out a bloom-filter containing all the elements in > the small table. > Each element from the big table is first searched in the bloom filter, and > only in case of a positive match, > the small table hash table is explored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2929) race condition in DAG execute tasks for hive
[ https://issues.apache.org/jira/browse/HIVE-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248114#comment-13248114 ] Namit Jain commented on HIVE-2929: -- https://reviews.facebook.net/differential/diff/8571/ > race condition in DAG execute tasks for hive > > > Key: HIVE-2929 > URL: https://issues.apache.org/jira/browse/HIVE-2929 > Project: Hive > Issue Type: Bug >Reporter: Namit Jain > > select ... > ( > SubQuery involving MapReduce > union all > SubQuery involving MapReduce > ); > or > select ... > (SubQuery involving MapReduce) > join > (SubQuery involving MapReduce) > ; > If both the subQueries finish at nearly the same time, there is a race > condition in which the results of the subQuery finishing last will be > completely missed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2929) race condition in DAG execute tasks for hive
race condition in DAG execute tasks for hive Key: HIVE-2929 URL: https://issues.apache.org/jira/browse/HIVE-2929 Project: Hive Issue Type: Bug Reporter: Namit Jain select ... ( SubQuery involving MapReduce union all SubQuery involving MapReduce ); or select ... (SubQuery involving MapReduce) join (SubQuery involving MapReduce) ; If both the subQueries finish at nearly the same time, there is a race condition in which the results of the subQuery finishing last will be completely missed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2901) Hive union with NULL constant and string in same column returns all null
[ https://issues.apache.org/jira/browse/HIVE-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2901: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Navis > Hive union with NULL constant and string in same column returns all null > > > Key: HIVE-2901 > URL: https://issues.apache.org/jira/browse/HIVE-2901 > Project: Hive > Issue Type: Bug >Reporter: Kevin Wilfong >Assignee: Navis >Priority: Critical > Attachments: HIVE-2901.D2541.1.patch, HIVE-2901.D2541.2.patch, > HIVE-2901.D2541.3.patch > > > select x from (select value as x from src union all select NULL as x from > src)a; > This query produces all nulls, where value is a string column. > Notably, > select x from (select key as x from src union all select NULL as x from src)a; > where key is a string, but can be cast to a double, the query returns correct > results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Debugging hive tasks?
you can try to set mapred.job.reuse.jvm.num.tasks to higher value in mapred-site.xml conf file, this will enable jvm reuse. 在 2012年4月2日 上午2:56,Travis Crawford 写道: > Hey hive gurus - > > On the wiki I found a section about running Hive jobs in local mode, > however, it uses a separate JVM. Is it possible to run the map reduce jobs > inside the current process? I'm able to attach a debugger to the CLI > process but haven't been able to figure out debugging the MR tasks. Any > tips would be appreciated (and I'll update the wiki). > > > https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-Hive%2CMapReduceandLocalMode > > Thanks! > Travis >
[jira] [Commented] (HIVE-1721) use bloom filters to improve the performance of joins
[ https://issues.apache.org/jira/browse/HIVE-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247985#comment-13247985 ] alex gemini commented on HIVE-1721: --- I'm wondering how we apply bloom filter to big table.we use map side join for small table < 25M, if we use bloom filter build small table,we maybe can increase small table size to 200M, but in big table map stage,we need to read bloom filter and writer intermediate result back to disk and then reading this intermediate result to check the real small table,we still can't hold the actual real small table into memory(correct the logic if I'm wrong),we pay the cost of writer a intermediate result which is very close to final result.In this case we can't increase the map number because it will double the penalty of io.I guess it will only get benefit in three table join on same join key,one small with 2 big.In my opinion the other db system can get benefit of bloom filter is because they can hold the intermediate result in memory for further processing (like oracle) or print it immediate (like hbase). > use bloom filters to improve the performance of joins > - > > Key: HIVE-1721 > URL: https://issues.apache.org/jira/browse/HIVE-1721 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain > Labels: gsoc, gsoc2012, optimization > > In case of map-joins, it is likely that the big table will not find many > matching rows from the small table. > Currently, we perform a hash-map lookup for every row in the big table, which > can be pretty expensive. > It might be useful to try out a bloom-filter containing all the elements in > the small table. > Each element from the big table is first searched in the bloom filter, and > only in case of a positive match, > the small table hash table is explored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-600) Running TPC-H queries on Hive
[ https://issues.apache.org/jira/browse/HIVE-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247949#comment-13247949 ] alex gemini commented on HIVE-600: -- I guess the Q9 failed is because hive set mapred.min.split.size=536870912, the default jvm setting for task launch is 200M, so it failed,remove this setting or set jvm memory for a larger value would be nice. > Running TPC-H queries on Hive > - > > Key: HIVE-600 > URL: https://issues.apache.org/jira/browse/HIVE-600 > Project: Hive > Issue Type: New Feature >Reporter: Yuntao Jia >Assignee: Yuntao Jia > Attachments: TPC-H_on_Hive_2009-08-11.pdf, > TPC-H_on_Hive_2009-08-11.tar.gz, TPC-H_on_Hive_2009-08-14.tar.gz > > > The goal is to run all TPC-H (http://www.tpc.org/tpch/) benchmark queries on > Hive for two reasons. First, through those queries, we would like to find the > new features that we need to put into Hive so that Hive supports common SQL > queries. Second, we would like to measure the performance of Hive to find out > what Hive is not good at. We can then improve Hive based on those > information. > For queries that are not supported now in Hive, I will try to rewrite them to > one or more Hive-supported queries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2909: -- Attachment: HIVE-2909.D2595.5.patch dikanggu updated the revision "HIVE-2909 [jira] SHOW COLUMNS table_name; to provide a comma-delimited list of columns.". Reviewers: JIRA, njain address Namit's comments, add the partition columns. REVISION DETAIL https://reviews.facebook.net/D2595 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java ql/src/java/org/apache/hadoop/hive/ql/plan/ShowColumnsDesc.java ql/src/test/queries/clientpositive/show_columns.q ql/src/test/results/clientpositive/show_columns.q.out > SHOW COLUMNS table_name; to provide a comma-delimited list of columns. > -- > > Key: HIVE-2909 > URL: https://issues.apache.org/jira/browse/HIVE-2909 > Project: Hive > Issue Type: New Feature >Reporter: Adam Kramer >Assignee: Dikang Gu >Priority: Minor > Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, > HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch, HIVE-2909.D2595.5.patch > > > Due to the way that SELECT * and partitioning works, it is frequently > obnoxious to insert data into tables of the same schema. > This could be fixed in a number of ways, all murky; this feature request > reduces the obnoxicity of the current situation. > SHOW COLUMNS foo; > OK > bar, baz, tball, ds > ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2928) Support for Oracle-backed Hive-Metastore ("longvarchar" to "clob" in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-2928: --- Attachment: HIVE-2928.patch Schema-creation in Oracle. + Changes to package.jdo ("longvarchar" to "clob"). > Support for Oracle-backed Hive-Metastore ("longvarchar" to "clob" in > package.jdo) > - > > Key: HIVE-2928 > URL: https://issues.apache.org/jira/browse/HIVE-2928 > Project: Hive > Issue Type: New Feature > Components: Metastore >Affects Versions: 0.9.0 >Reporter: Mithun Radhakrishnan > Attachments: HIVE-2928.patch > > > I'm trying to get the Hive-Metastore to work when backed by an Oracle > backend. There's a change to hive's package.jdo that I'd like advice/comments > on. > One sticking point on working with Oracle has been the TBLS table (MTable) > and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). > Oracle doesn't support more than one LONGVARCHAR property per table (for > reason of legacy), and prefers that one use CLOBs instead. If one switches to > CLOB properties, with no modification to hive's package.jdo, one sees the > following exception: > > Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB > (datastore), but type expected was LONGVARCHAR (metadata). Please check that > the type in the datastore and the type specified in the MetaData are > consistent. > org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: > Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB > (datastore), but type expected was LONGVARCHAR (metadata). Please check that > the type in the datastore and the type specified in the MetaData are > consistent. > at > org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) > at > org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 > > But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of > LONGVARCHARs, things look promising: > 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet > to scale-test.) > 2. These modified hive-libraries work as is with pre-existing mysql > metastores. Migrating data isn't a worry. > 3. The unit-tests seem to run through. > Would there be opposition to changing the package.jdo's LONGVARCHAR > references to CLOB, if this works with mysql and with Oracle? > Mithun > P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm > testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2928) Support for Oracle-backed Hive-Metastore ("longvarchar" to "clob" in package.jdo)
[ https://issues.apache.org/jira/browse/HIVE-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated HIVE-2928: --- Affects Version/s: (was: 0.8.1) 0.9.0 > Support for Oracle-backed Hive-Metastore ("longvarchar" to "clob" in > package.jdo) > - > > Key: HIVE-2928 > URL: https://issues.apache.org/jira/browse/HIVE-2928 > Project: Hive > Issue Type: New Feature > Components: Metastore >Affects Versions: 0.9.0 >Reporter: Mithun Radhakrishnan > Attachments: HIVE-2928.patch > > > I'm trying to get the Hive-Metastore to work when backed by an Oracle > backend. There's a change to hive's package.jdo that I'd like advice/comments > on. > One sticking point on working with Oracle has been the TBLS table (MTable) > and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). > Oracle doesn't support more than one LONGVARCHAR property per table (for > reason of legacy), and prefers that one use CLOBs instead. If one switches to > CLOB properties, with no modification to hive's package.jdo, one sees the > following exception: > > Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB > (datastore), but type expected was LONGVARCHAR (metadata). Please check that > the type in the datastore and the type specified in the MetaData are > consistent. > org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: > Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB > (datastore), but type expected was LONGVARCHAR (metadata). Please check that > the type in the datastore and the type specified in the MetaData are > consistent. > at > org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) > at > org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 > > But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of > LONGVARCHARs, things look promising: > 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet > to scale-test.) > 2. These modified hive-libraries work as is with pre-existing mysql > metastores. Migrating data isn't a worry. > 3. The unit-tests seem to run through. > Would there be opposition to changing the package.jdo's LONGVARCHAR > references to CLOB, if this works with mysql and with Oracle? > Mithun > P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm > testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2928) Support for Oracle-backed Hive-Metastore ("longvarchar" to "clob" in package.jdo)
Support for Oracle-backed Hive-Metastore ("longvarchar" to "clob" in package.jdo) - Key: HIVE-2928 URL: https://issues.apache.org/jira/browse/HIVE-2928 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.8.1 Reporter: Mithun Radhakrishnan I'm trying to get the Hive-Metastore to work when backed by an Oracle backend. There's a change to hive's package.jdo that I'd like advice/comments on. One sticking point on working with Oracle has been the TBLS table (MTable) and its 2 LONGVARCHAR properties (VIEW_ORIGINAL_TEXT and VIEW_EXPANDED_TEXT). Oracle doesn't support more than one LONGVARCHAR property per table (for reason of legacy), and prefers that one use CLOBs instead. If one switches to CLOB properties, with no modification to hive's package.jdo, one sees the following exception: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. org.datanucleus.store.rdbms.exceptions.IncompatibleDataTypeException: Incompatible data type for column TBLS.VIEW_EXPANDED_TEXT : was CLOB (datastore), but type expected was LONGVARCHAR (metadata). Please check that the type in the datastore and the type specified in the MetaData are consistent. at org.datanucleus.store.rdbms.table.ColumnImpl.validate(ColumnImpl.java:521) at org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:2 But if one rebuilds Hive with the package.jdo changed to use CLOBs instead of LONGVARCHARs, things look promising: 1. The exception no longer occurs. Things seem to work with Oracle. (I've yet to scale-test.) 2. These modified hive-libraries work as is with pre-existing mysql metastores. Migrating data isn't a worry. 3. The unit-tests seem to run through. Would there be opposition to changing the package.jdo's LONGVARCHAR references to CLOB, if this works with mysql and with Oracle? Mithun P.S. I also have a working hive-schema-0.9.0-oracle.sql script that I'm testing, for the related issue of creating the required tables in Oracle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247891#comment-13247891 ] Carl Steinbach commented on HIVE-2909: -- MySQL has a SHOW COLUMNS command: http://dev.mysql.com/doc/refman/5.0/en/show-columns.html Seems like we should follow convention and try to duplicate that instead of inventing our own behavior. Also, I think returning each column name in a separate row is much more convenient (and consistent) for clients than forcing them to tokenize the names by splitting on ',' > SHOW COLUMNS table_name; to provide a comma-delimited list of columns. > -- > > Key: HIVE-2909 > URL: https://issues.apache.org/jira/browse/HIVE-2909 > Project: Hive > Issue Type: New Feature >Reporter: Adam Kramer >Assignee: Dikang Gu >Priority: Minor > Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, > HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch > > > Due to the way that SELECT * and partitioning works, it is frequently > obnoxious to insert data into tables of the same schema. > This could be fixed in a number of ways, all murky; this feature request > reduces the obnoxicity of the current situation. > SHOW COLUMNS foo; > OK > bar, baz, tball, ds > ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2585) Collapse hive.metastore.uris and hive.metastore.local
[ https://issues.apache.org/jira/browse/HIVE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247885#comment-13247885 ] Phabricator commented on HIVE-2585: --- thw has commented on the revision "HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local". INLINE COMMENTS conf/hive-default.xml.template:102 A scenario where this would lead to change of existing behavior is when client and metastore server point to the same conf, with client using the thrift URI and server javax.jdo.option.ConnectionURL (should only happen in dev environments). REVISION DETAIL https://reviews.facebook.net/D2559 BRANCH svn > Collapse hive.metastore.uris and hive.metastore.local > - > > Key: HIVE-2585 > URL: https://issues.apache.org/jira/browse/HIVE-2585 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-2585.D2559.1.patch > > > We should just have hive.metastore.uris. If it is empty, we shall assume > local mode, if non-empty we shall use that string to connect to remote > metastore. Having two different keys for same information is confusing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2585) Collapse hive.metastore.uris and hive.metastore.local
[ https://issues.apache.org/jira/browse/HIVE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247875#comment-13247875 ] Phabricator commented on HIVE-2585: --- cwsteinbach has commented on the revision "HIVE-2585 [jira] Collapse hive.metastore.uris and hive.metastore.local". INLINE COMMENTS conf/hive-default.xml.template:102 HiveConf.init() seems like the right place to make these checks. REVISION DETAIL https://reviews.facebook.net/D2559 BRANCH svn > Collapse hive.metastore.uris and hive.metastore.local > - > > Key: HIVE-2585 > URL: https://issues.apache.org/jira/browse/HIVE-2585 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-2585.D2559.1.patch > > > We should just have hive.metastore.uris. If it is empty, we shall assume > local mode, if non-empty we shall use that string to connect to remote > metastore. Having two different keys for same information is confusing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247860#comment-13247860 ] Ashutosh Chauhan commented on HIVE-2923: I was able to reproduce failures before the patch. With the latest patch, failures went away and tests pass. Patch looks good to me. Those who were seeing failures earlier can try with this patch and report back. > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > Fix For: 0.9.0 > > Attachments: HIVE-2923.patch > > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247862#comment-13247862 ] Ashutosh Chauhan commented on HIVE-2923: Thanks, Thomas for taking this up. Very much appreciated! > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > Fix For: 0.9.0 > > Attachments: HIVE-2923.patch > > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247852#comment-13247852 ] Phabricator commented on HIVE-2926: --- cwsteinbach has requested changes to the revision "HIVE-2926 [jira] Expose some information about the metastore through JMX". INLINE COMMENTS common/src/java/org/apache/hadoop/hive/common/HiveVersionAnnotation.java:26 Comment should reference Hive instead of Hadoop. Same thing applies to several of the comments below. common/src/java/org/apache/hadoop/hive/common/VersionInfo.java:101 Change to "Apache Hive" ql/src/test/results/clientnegative/column_rename1.q.out:26 These q file updates don't belong here, right? common/build.xml:30 I think this target should be moved into build-common.xml and should get called for all sub-projects. This will make it a lot easier to diagnose cases where people are using JARs from different versions of Hive. common/src/java/org/apache/hadoop/hive/common/HiveVersionAnnotation.java:50 According to saveVersion.sh this could also return the URL for a Git repo. The comment should be updated to reflect this, and I think the name of the method should be changed to "getRepositoryUrl()" common/src/java/org/apache/hadoop/hive/common/HiveVersionAnnotation.java:55 Ditto. common/src/java/org/apache/hadoop/hive/common/VersionInfo.java:75 "source code repository URL" common/src/java/org/apache/hadoop/hive/common/VersionInfo.java:95 Does it make sense to reference a revision number without also mentioning the repository URL? common/src/java/org/apache/hadoop/hive/common/VersionInfo.java:102 This references svn. What if this was built using git? metastore/src/java/org/apache/hadoop/hive/metastore/MXBeanImpl.java:33 This assumes that VersionInfo.getRevision() returns an SVN revision number. metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:3060 Does it make sense to include minWorkerThreads, or the value of hive.metastore.uris? What about the value of useSasl or tcpKeepAlive? REVISION DETAIL https://reviews.facebook.net/D2637 BRANCH HIVE-2926-branch > Expose some information about the metastore through JMX > --- > > Key: HIVE-2926 > URL: https://issues.apache.org/jira/browse/HIVE-2926 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.9.0 > > Attachments: HIVE-2926.D2637.1.patch, hive-2926.1.patch, > metastore-jmx.patch > > > Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247840#comment-13247840 ] Phabricator commented on HIVE-2926: --- ddraj has commented on the revision "HIVE-2926 [jira] Expose some information about the metastore through JMX". Ignore the two patch changes in *.out files. I don't know how these crept in. REVISION DETAIL https://reviews.facebook.net/D2637 > Expose some information about the metastore through JMX > --- > > Key: HIVE-2926 > URL: https://issues.apache.org/jira/browse/HIVE-2926 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.9.0 > > Attachments: HIVE-2926.D2637.1.patch, hive-2926.1.patch, > metastore-jmx.patch > > > Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2901) Hive union with NULL constant and string in same column returns all null
[ https://issues.apache.org/jira/browse/HIVE-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247829#comment-13247829 ] Phabricator commented on HIVE-2901: --- njain has accepted the revision "HIVE-2901 [jira] Hive union with NULL constant and string in same column returns all null". REVISION DETAIL https://reviews.facebook.net/D2541 BRANCH DPAL-1029 > Hive union with NULL constant and string in same column returns all null > > > Key: HIVE-2901 > URL: https://issues.apache.org/jira/browse/HIVE-2901 > Project: Hive > Issue Type: Bug >Reporter: Kevin Wilfong >Assignee: Navis >Priority: Critical > Attachments: HIVE-2901.D2541.1.patch, HIVE-2901.D2541.2.patch, > HIVE-2901.D2541.3.patch > > > select x from (select value as x from src union all select NULL as x from > src)a; > This query produces all nulls, where value is a string column. > Notably, > select x from (select key as x from src union all select NULL as x from src)a; > where key is a string, but can be cast to a double, the query returns correct > results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247820#comment-13247820 ] Phabricator commented on HIVE-2926: --- ddraj has added reviewers to the revision "HIVE-2926 [jira] Expose some information about the metastore through JMX". Added Reviewers: ashutoshc, khorgath, cwsteinbach REVISION DETAIL https://reviews.facebook.net/D2637 > Expose some information about the metastore through JMX > --- > > Key: HIVE-2926 > URL: https://issues.apache.org/jira/browse/HIVE-2926 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.9.0 > > Attachments: HIVE-2926.D2637.1.patch, hive-2926.1.patch, > metastore-jmx.patch > > > Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Attachment: HIVE-2923.patch Updated patch to increase timeout for slow test environments. > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > Fix For: 0.9.0 > > Attachments: HIVE-2923.patch > > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2926: -- Attachment: HIVE-2926.D2637.1.patch ddraj requested code review of "HIVE-2926 [jira] Expose some information about the metastore through JMX". Reviewers: JIRA Adds VersionInfo and makes some metastore information exposed through JMX Expose some information about the metastore through JMX TEST PLAN Did manual testing of the JMX stuff (including the version). Accessed the data through jconsole. REVISION DETAIL https://reviews.facebook.net/D2637 AFFECTED FILES common/build.xml common/saveVersion.sh common/src/java/org/apache/hadoop/hive/common/HiveVersionAnnotation.java common/src/java/org/apache/hadoop/hive/common/VersionInfo.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/MXBean.java metastore/src/java/org/apache/hadoop/hive/metastore/MXBeanImpl.java ql/src/test/results/clientnegative/column_rename1.q.out ql/src/test/results/clientnegative/column_rename4.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6063/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Expose some information about the metastore through JMX > --- > > Key: HIVE-2926 > URL: https://issues.apache.org/jira/browse/HIVE-2926 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.9.0 > > Attachments: HIVE-2926.D2637.1.patch, hive-2926.1.patch, > metastore-jmx.patch > > > Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Attachment: (was: HIVE-2923.patch) > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > Fix For: 0.9.0 > > Attachments: HIVE-2923.patch > > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2863) Ambiguous table name or column reference message displays when table and column names are the same
[ https://issues.apache.org/jira/browse/HIVE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2863: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Navis > Ambiguous table name or column reference message displays when table and > column names are the same > -- > > Key: HIVE-2863 > URL: https://issues.apache.org/jira/browse/HIVE-2863 > Project: Hive > Issue Type: Bug >Reporter: Mauro Cazzari >Assignee: Navis > Attachments: HIVE-2863.D2361.1.patch > > > Given the following table: > CREATE TABLE `Y` (`y` DOUBLE) ROW FORMAT DELIMITED FIELDS TERMINATED BY > '\001' STORED AS TEXTFILE; > The following query fails: > SELECT `Y`.`y` FROM `Y` WHERE ( `y` = 1 ) > ERROR: java.sql.SQLException: Query returned non-zero code: 10, cause: > FAILED: Error in >semantic analysis: Line 1:36 Ambiguous table alias or column reference > '`y`' > ERROR: Unable to execute Hadoop query. > ERROR: Prepare error. SQL statement: SELECT `Y`.`y` FROM `Y` WHERE ( `y` > = 1 ). > The problem goes away if the table and column names do not match. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247784#comment-13247784 ] Carl Steinbach commented on HIVE-2926: -- Sounds good. Please submit a phabricator request (https://cwiki.apache.org/Hive/phabricatorcodereview.html) when the patch is ready for review. > Expose some information about the metastore through JMX > --- > > Key: HIVE-2926 > URL: https://issues.apache.org/jira/browse/HIVE-2926 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.9.0 > > Attachments: hive-2926.1.patch, metastore-jmx.patch > > > Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HIVE-2926: -- Attachment: hive-2926.1.patch @Ashutosh the patch attached here has been tested. Yes this assumes executables like 'sed'. I don't have a better solution at this point for handling that issue. The way version is derived in this patch is similar to the way Hadoop/HBase does.. @Carl, I changed the 'sh' to 'bash' > Expose some information about the metastore through JMX > --- > > Key: HIVE-2926 > URL: https://issues.apache.org/jira/browse/HIVE-2926 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.9.0 > > Attachments: hive-2926.1.patch, metastore-jmx.patch > > > Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247771#comment-13247771 ] Phabricator commented on HIVE-2909: --- njain has commented on the revision "HIVE-2909 [jira] SHOW COLUMNS table_name; to provide a comma-delimited list of columns.". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:2039 for partitioned tables, all partitioned columns also cols.addAll(tbl.getPartCols()); REVISION DETAIL https://reviews.facebook.net/D2595 > SHOW COLUMNS table_name; to provide a comma-delimited list of columns. > -- > > Key: HIVE-2909 > URL: https://issues.apache.org/jira/browse/HIVE-2909 > Project: Hive > Issue Type: New Feature >Reporter: Adam Kramer >Assignee: Dikang Gu >Priority: Minor > Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, > HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch > > > Due to the way that SELECT * and partitioning works, it is frequently > obnoxious to insert data into tables of the same schema. > This could be fixed in a number of ways, all murky; this feature request > reduces the obnoxicity of the current situation. > SHOW COLUMNS foo; > OK > bar, baz, tball, ds > ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247759#comment-13247759 ] Carl Steinbach commented on HIVE-2926: -- bin/hive uses Bash. I think we should do the same thing here. Also note that on most platforms /bin/sh actually maps to bash, or to something entirely different (e.g. https://wiki.ubuntu.com/DashAsBinSh). > Expose some information about the metastore through JMX > --- > > Key: HIVE-2926 > URL: https://issues.apache.org/jira/browse/HIVE-2926 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.9.0 > > Attachments: metastore-jmx.patch > > > Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2926) Expose some information about the metastore through JMX
[ https://issues.apache.org/jira/browse/HIVE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247743#comment-13247743 ] Ashutosh Chauhan commented on HIVE-2926: Couple of comments: * This assumes sh, cat, sed etc. to be available in the environment. So, it may not work where these tools are not present, which is fine but thought to point this out. I tested on mac and linux, it builds fine. * I don't see how your generated package-info.java will get compiled. You create a .java file in build dir. Have you tested that it works as you expect? > Expose some information about the metastore through JMX > --- > > Key: HIVE-2926 > URL: https://issues.apache.org/jira/browse/HIVE-2926 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.9.0 > > Attachments: metastore-jmx.patch > > > Expose some information about the metastore through JMX -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2909: -- Attachment: HIVE-2909.D2595.4.patch dikanggu updated the revision "HIVE-2909 [jira] SHOW COLUMNS table_name; to provide a comma-delimited list of columns.". Reviewers: JIRA, njain address Namit's comments. REVISION DETAIL https://reviews.facebook.net/D2595 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java ql/src/java/org/apache/hadoop/hive/ql/plan/ShowColumnsDesc.java ql/src/test/queries/clientpositive/show_columns.q ql/src/test/results/clientpositive/show_columns.q.out > SHOW COLUMNS table_name; to provide a comma-delimited list of columns. > -- > > Key: HIVE-2909 > URL: https://issues.apache.org/jira/browse/HIVE-2909 > Project: Hive > Issue Type: New Feature >Reporter: Adam Kramer >Assignee: Dikang Gu >Priority: Minor > Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, > HIVE-2909.D2595.3.patch, HIVE-2909.D2595.4.patch > > > Due to the way that SELECT * and partitioning works, it is frequently > obnoxious to insert data into tables of the same schema. > This could be fixed in a number of ways, all murky; this feature request > reduces the obnoxicity of the current situation. > SHOW COLUMNS foo; > OK > bar, baz, tball, ds > ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2909) SHOW COLUMNS table_name; to provide a comma-delimited list of columns.
[ https://issues.apache.org/jira/browse/HIVE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247679#comment-13247679 ] Phabricator commented on HIVE-2909: --- njain has commented on the revision "HIVE-2909 [jira] SHOW COLUMNS table_name; to provide a comma-delimited list of columns.". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:2037 dont sort the columns. Use the same order as used by describeTable REVISION DETAIL https://reviews.facebook.net/D2595 > SHOW COLUMNS table_name; to provide a comma-delimited list of columns. > -- > > Key: HIVE-2909 > URL: https://issues.apache.org/jira/browse/HIVE-2909 > Project: Hive > Issue Type: New Feature >Reporter: Adam Kramer >Assignee: Dikang Gu >Priority: Minor > Attachments: HIVE-2909.D2595.1.patch, HIVE-2909.D2595.2.patch, > HIVE-2909.D2595.3.patch > > > Due to the way that SELECT * and partitioning works, it is frequently > obnoxious to insert data into tables of the same schema. > This could be fixed in a number of ways, all murky; this feature request > reduces the obnoxicity of the current situation. > SHOW COLUMNS foo; > OK > bar, baz, tball, ds > ...then I could just copy the first three and not the last. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-0.8.1-SNAPSHOT-h0.21 - Build # 244 - Failure
Changes for Build #244 1 tests failed. REGRESSION: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:9440) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) The Apache Jenkins build system has built Hive-0.8.1-SNAPSHOT-h0.21 (build #244) Status: Failure Check console output at https://builds.apache.org/job/Hive-0.8.1-SNAPSHOT-h0.21/244/ to view the results.
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[ https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean McNamara updated HIVE-2927: Description: *Background:* get_json_object extracts json objects from a json string based on a specified path. *Problem:* The current implementation of get_json_object can't see keys with a '.' in them. Our data contains '.' in the keys, so we have to filter our json keys through a streaming script to replace '.' for '_'. *Example:* {{json = {"a":{"b": 1}, "c.d": 2}}} {{get_json_object(json, "$.a.b") returns: 1}} {{get_json_object(json, "$.c.d") returns: NULL}} In the present implementation of get_json_object, c.d is not addressable. *Proposal:* The desired behavior would be to allow the JSON path to be escape-able, like so: {{get_json_object(json, '$.c\\\.d') would return: 2}} was: *Background:* get_json_object extracts json objects from a json string based on a specified path. *Problem:* The current implementation of get_json_object can't see keys with a '.' in them. Our data contains '.' in the keys, so we have to filter our json keys through a streaming script to replace '.' for '_'. *Example:* {{json = {"a":{"b": 1}, "c.d": 2}}} {{get_json_object(json, "a.b") returns: 1}} {{get_json_object(json, "c.d") returns: NULL}} In the present implementation of get_json_object, c.d is not addressable. *Proposal:* The desired behavior would be to allow the JSON path to be escape-able, like so: {{get_json_object(json, '$.c\\\.d') would return: 2}} Affects Version/s: 0.8.1 Fix Version/s: 0.9.0 > Allow escape character in get_json_object > - > > Key: HIVE-2927 > URL: https://issues.apache.org/jira/browse/HIVE-2927 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 0.8.1 >Reporter: Sean McNamara > Fix For: 0.9.0 > > Attachments: HIVE-2927.1.patch.txt > > Original Estimate: 0h > Remaining Estimate: 0h > > *Background:* > get_json_object extracts json objects from a json string based on a specified > path. > *Problem:* > The current implementation of get_json_object can't see keys with a '.' in > them. Our data contains '.' in the keys, so we have to filter our json keys > through a streaming script to replace '.' for '_'. > *Example:* > {{json = {"a":{"b": 1}, "c.d": 2}}} > {{get_json_object(json, "$.a.b") returns: 1}} > {{get_json_object(json, "$.c.d") returns: NULL}} > In the present implementation of get_json_object, c.d is not addressable. > *Proposal:* > The desired behavior would be to allow the JSON path to be escape-able, like > so: > {{get_json_object(json, '$.c\\\.d') would return: 2}} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2822) Add JSON output to the hive ddl commands
[ https://issues.apache.org/jira/browse/HIVE-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247537#comment-13247537 ] Hudson commented on HIVE-2822: -- Integrated in Hive-trunk-h0.21 #1354 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1354/]) HIVE-2822: Second part of it (ctdean via Ashutosh Chauhan) (Revision 1309667) HIVE-2822 [jira] Add JSON output to the hive ddl commands (Chris Dean via Ashutosh Chauhan) Summary: JSON output for DDL commands By setting the variable hive.format=json then ddl commands will produce json output suitable for client side parsing. For example, to list all the databases one might get: { "databases": [ "default" ] } The goal is to have an option to produce JSON output of the DDL commands that is easily machine parseable. For example, "desc my_table" currently gives idbigint user string and we want to allow a json output: { "columns": [ {"name": "id", "type": "bigint"}, {"name": "user", "type": "string"} ] } Test Plan: Run the provided unit tests Reviewers: JIRA, ashutoshc Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D2475 (Revision 1309666) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309667 Files : * /hive/trunk/ql/src/test/results/clientnegative/column_rename1.q.out * /hive/trunk/ql/src/test/results/clientnegative/column_rename4.q.out hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309666 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MapBuilder.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatter.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java * /hive/trunk/ql/src/test/queries/clientpositive/describe_database_json.q * /hive/trunk/ql/src/test/queries/clientpositive/describe_table_json.q * /hive/trunk/ql/src/test/queries/clientpositive/misc_json.q * /hive/trunk/ql/src/test/queries/clientpositive/partitions_json.q * /hive/trunk/ql/src/test/results/clientnegative/database_create_already_exists.q.out * /hive/trunk/ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out * /hive/trunk/ql/src/test/results/clientpositive/describe_database_json.q.out * /hive/trunk/ql/src/test/results/clientpositive/describe_table_json.q.out * /hive/trunk/ql/src/test/results/clientpositive/misc_json.q.out * /hive/trunk/ql/src/test/results/clientpositive/partitions_json.q.out > Add JSON output to the hive ddl commands > > > Key: HIVE-2822 > URL: https://issues.apache.org/jira/browse/HIVE-2822 > Project: Hive > Issue Type: Improvement >Reporter: Chris Dean >Assignee: Chris Dean > Fix For: 0.9.0 > > Attachments: HIVE-2822.03-branch0-8.patch, HIVE-2822.03.patch, > HIVE-2822.03b.patch, HIVE-2822.04-branch-08.patch, > HIVE-2822.05-branch0-8-1.patch, HIVE-2822.05-branch0-8.patch, > HIVE-2822.05.patch, HIVE-2822.D2475.1.patch, HIVE-2822.D2475.2.patch, > HIVE-2822.D2475.3.patch, HIVE-2822.D2475.4.patch, > hive-json-01-branch0-8.patch, hive-json-01.patch, > hive-json-02-branch0-8.patch, hive-json-02.patch > > > The goal is to have an option to produce JSON output of the DDL commands that > is easily machine parseable. > For example, "desc my_table" currently gives > {noformat} > idbigint > user string > {noformat} > and we want to allow a json output: > {noformat} > { > "columns": [ > {"name": "id", "type": "bigint"}, > {"name": "user", "type": "string"} > ] > } > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2559) Add target to install Hive JARs/POMs in the local Maven cache
[ https://issues.apache.org/jira/browse/HIVE-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247535#comment-13247535 ] Hudson commented on HIVE-2559: -- Integrated in Hive-trunk-h0.21 #1354 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1354/]) HIVE-2559 : Add target to install Hive JARs/POMs in the local Maven cache (Alan Gates via Ashutosh Chauhan) (Revision 1309675) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309675 Files : * /hive/trunk/build.xml > Add target to install Hive JARs/POMs in the local Maven cache > - > > Key: HIVE-2559 > URL: https://issues.apache.org/jira/browse/HIVE-2559 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Affects Versions: 0.9.0 >Reporter: Alejandro Abdelnur >Assignee: Alan Gates >Priority: Critical > Fix For: 0.9.0 > > Attachments: HIVE-2559.patch > > > HIVE-2391 is producing usable Maven artifacts. > However, it only as a target to deploy/publish those artifacts to Apache > Maven repos. > There should be a new target to locally install Hive Maven artifacts, thus > enabling their use from other projects before they are committed/publish to > Apache Maven (this is critical to test patches that may address issues in > downstream components). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2879) Add "rat" target to build to look for missing license headers
[ https://issues.apache.org/jira/browse/HIVE-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247536#comment-13247536 ] Hudson commented on HIVE-2879: -- Integrated in Hive-trunk-h0.21 #1354 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1354/]) HIVE-2879 : Add rat target to build to look for missing license headers (Owen Omalley via Ashutosh Chauhan) (Revision 1309676) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309676 Files : * /hive/trunk/build.xml * /hive/trunk/ivy.xml * /hive/trunk/ivy/libraries.properties > Add "rat" target to build to look for missing license headers > - > > Key: HIVE-2879 > URL: https://issues.apache.org/jira/browse/HIVE-2879 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 0.9.0 > > Attachments: HIVE-2879.D2373.1.patch, HIVE-2879.D2373.2.patch, > HIVE-2879.D2373.3.patch > > > We should have a "rat" target that can generate an Apache RAT report. RAT is > used for finding missing license headers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1354 - Failure
Changes for Build #1354 [hashutosh] HIVE-2879 : Add rat target to build to look for missing license headers (Owen Omalley via Ashutosh Chauhan) [hashutosh] HIVE-2559 : Add target to install Hive JARs/POMs in the local Maven cache (Alan Gates via Ashutosh Chauhan) [hashutosh] HIVE-2822: Second part of it (ctdean via Ashutosh Chauhan) [hashutosh] HIVE-2822 [jira] Add JSON output to the hive ddl commands (Chris Dean via Ashutosh Chauhan) Summary: JSON output for DDL commands By setting the variable hive.format=json then ddl commands will produce json output suitable for client side parsing. For example, to list all the databases one might get: { "databases": [ "default" ] } The goal is to have an option to produce JSON output of the DDL commands that is easily machine parseable. For example, "desc my_table" currently gives idbigint user string and we want to allow a json output: { "columns": [ {"name": "id", "type": "bigint"}, {"name": "user", "type": "string"} ] } Test Plan: Run the provided unit tests Reviewers: JIRA, ashutoshc Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D2475 1 tests failed. REGRESSION: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. at junit.framework.Assert.fail(Assert.java:47) at org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:10418) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:154) at junit.framework.TestCase.runBare(TestCase.java:127) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785) The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1354) Status: Failure Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1354/ to view the results.
[jira] [Updated] (HIVE-2711) Make the header of RCFile unique
[ https://issues.apache.org/jira/browse/HIVE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-2711: Attachment: rc-file-v0.rc Here's the binary file that arc didn't include in the patch. > Make the header of RCFile unique > > > Key: HIVE-2711 > URL: https://issues.apache.org/jira/browse/HIVE-2711 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-2711.D2115.1.patch, HIVE-2711.D2115.2.patch, > HIVE-2711.D2115.3.patch, HIVE-2711.D2571.1.patch, rc-file-v0.rc > > > The RCFile implementation was copied from Hadoop's SequenceFile and copied > the 'magic' string in the header. This means that you can't use the header to > distinguish between RCFiles and SequenceFiles. > I'd propose that we create a new header for RCFiles (RCF?) to replace the > current SEQ. To maintain compatibility, we'll need to continue to accept the > current 'SEQ\06' and just make new files contain the new header. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2711) Make the header of RCFile unique
[ https://issues.apache.org/jira/browse/HIVE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247463#comment-13247463 ] Phabricator commented on HIVE-2711: --- ashutoshc has accepted the revision "HIVE-2711 [jira] Make the header of RCFile unique". +1 REVISION DETAIL https://reviews.facebook.net/D2115 BRANCH h-2711 > Make the header of RCFile unique > > > Key: HIVE-2711 > URL: https://issues.apache.org/jira/browse/HIVE-2711 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-2711.D2115.1.patch, HIVE-2711.D2115.2.patch, > HIVE-2711.D2115.3.patch, HIVE-2711.D2571.1.patch > > > The RCFile implementation was copied from Hadoop's SequenceFile and copied > the 'magic' string in the header. This means that you can't use the header to > distinguish between RCFiles and SequenceFiles. > I'd propose that we create a new header for RCFiles (RCF?) to replace the > current SEQ. To maintain compatibility, we'll need to continue to accept the > current 'SEQ\06' and just make new files contain the new header. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[ https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean McNamara updated HIVE-2927: Component/s: (was: Security) Serializers/Deserializers > Allow escape character in get_json_object > - > > Key: HIVE-2927 > URL: https://issues.apache.org/jira/browse/HIVE-2927 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Sean McNamara > Attachments: HIVE-2927.1.patch.txt > > Original Estimate: 0h > Remaining Estimate: 0h > > *Background:* > get_json_object extracts json objects from a json string based on a specified > path. > *Problem:* > The current implementation of get_json_object can't see keys with a '.' in > them. Our data contains '.' in the keys, so we have to filter our json keys > through a streaming script to replace '.' for '_'. > *Example:* > {{json = {"a":{"b": 1}, "c.d": 2}}} > {{get_json_object(json, "a.b") returns: 1}} > {{get_json_object(json, "c.d") returns: NULL}} > In the present implementation of get_json_object, c.d is not addressable. > *Proposal:* > The desired behavior would be to allow the JSON path to be escape-able, like > so: > {{get_json_object(json, '$.c\\\.d') would return: 2}} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[ https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean McNamara updated HIVE-2927: Needs code review. thnx! > Allow escape character in get_json_object > - > > Key: HIVE-2927 > URL: https://issues.apache.org/jira/browse/HIVE-2927 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Sean McNamara > Attachments: HIVE-2927.1.patch.txt > > Original Estimate: 0h > Remaining Estimate: 0h > > *Background:* > get_json_object extracts json objects from a json string based on a specified > path. > *Problem:* > The current implementation of get_json_object can't see keys with a '.' in > them. Our data contains '.' in the keys, so we have to filter our json keys > through a streaming script to replace '.' for '_'. > *Example:* > {{json = {"a":{"b": 1}, "c.d": 2}}} > {{get_json_object(json, "a.b") returns: 1}} > {{get_json_object(json, "c.d") returns: NULL}} > In the present implementation of get_json_object, c.d is not addressable. > *Proposal:* > The desired behavior would be to allow the JSON path to be escape-able, like > so: > {{get_json_object(json, '$.c\\\.d') would return: 2}} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[ https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean McNamara updated HIVE-2927: Attachment: HIVE-2927.1.patch.txt Patch adds ability to escape '.' in JSON keys using \\. > Allow escape character in get_json_object > - > > Key: HIVE-2927 > URL: https://issues.apache.org/jira/browse/HIVE-2927 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Sean McNamara > Attachments: HIVE-2927.1.patch.txt > > Original Estimate: 0h > Remaining Estimate: 0h > > *Background:* > get_json_object extracts json objects from a json string based on a specified > path. > *Problem:* > The current implementation of get_json_object can't see keys with a '.' in > them. Our data contains '.' in the keys, so we have to filter our json keys > through a streaming script to replace '.' for '_'. > *Example:* > {{json = {"a":{"b": 1}, "c.d": 2}}} > {{get_json_object(json, "a.b") returns: 1}} > {{get_json_object(json, "c.d") returns: NULL}} > In the present implementation of get_json_object, c.d is not addressable. > *Proposal:* > The desired behavior would be to allow the JSON path to be escape-able, like > so: > {{get_json_object(json, '$.c\\\.d') would return: 2}} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[ https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean McNamara updated HIVE-2927: Description: *Background:* get_json_object extracts json objects from a json string based on a specified path. *Problem:* The current implementation of get_json_object can't see keys with a '.' in them. Our data contains '.' in the keys, so we have to filter our json keys through a streaming script to replace '.' for '_'. *Example:* {{json = {"a":{"b": 1}, "c.d": 2}}} {{get_json_object(json, "a.b") returns: 1}} {{get_json_object(json, "c.d") returns: NULL}} In the present implementation of get_json_object, c.d is not addressable. *Proposal:* The desired behavior would be to allow the JSON path to be escape-able, like so: {{get_json_object(json, '$.c\\\.d') would return: 2}} was: *Background:* get_json_object extracts json objects from a json string based on a specified path. *Problem:* The current implementation of get_json_object can't see keys with a '.' in them. Our data contains '.' in the keys, so we have to filter our json keys through a streaming script to replace '.' for '_'. *Example:* {{json = {"a":{"b": 1}, "c.d": 2}}} {{get_json_object(json, "a.b") returns: 1}} {{get_json_object(json, "c.d") returns: NULL}} In the present implementation of get_json_object, c.d is not addressable. *Proposal:* The desired behavior would be to allow the JSON path to be escape-able, like so: {{get_json_object(json, '$.c.d') would return: 2}} > Allow escape character in get_json_object > - > > Key: HIVE-2927 > URL: https://issues.apache.org/jira/browse/HIVE-2927 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Sean McNamara > Original Estimate: 0h > Remaining Estimate: 0h > > *Background:* > get_json_object extracts json objects from a json string based on a specified > path. > *Problem:* > The current implementation of get_json_object can't see keys with a '.' in > them. Our data contains '.' in the keys, so we have to filter our json keys > through a streaming script to replace '.' for '_'. > *Example:* > {{json = {"a":{"b": 1}, "c.d": 2}}} > {{get_json_object(json, "a.b") returns: 1}} > {{get_json_object(json, "c.d") returns: NULL}} > In the present implementation of get_json_object, c.d is not addressable. > *Proposal:* > The desired behavior would be to allow the JSON path to be escape-able, like > so: > {{get_json_object(json, '$.c\\\.d') would return: 2}} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2927) Allow escape character in get_json_object
[ https://issues.apache.org/jira/browse/HIVE-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean McNamara updated HIVE-2927: Component/s: (was: Serializers/Deserializers) Security Description: *Background:* get_json_object extracts json objects from a json string based on a specified path. *Problem:* The current implementation of get_json_object can't see keys with a '.' in them. Our data contains '.' in the keys, so we have to filter our json keys through a streaming script to replace '.' for '_'. *Example:* {{json = {"a":{"b": 1}, "c.d": 2}}} {{get_json_object(json, "a.b") returns: 1}} {{get_json_object(json, "c.d") returns: NULL}} In the present implementation of get_json_object, c.d is not addressable. *Proposal:* The desired behavior would be to allow the JSON path to be escape-able, like so: {{get_json_object(json, '$.c.d') would return: 2}} was: *Background:* get_json_object extracts json objects from a json string based on a specified path. *Problem:* The current implementation of get_json_object can't see keys with a '.' in them. Our data contains '.' in the keys, so we have to filter our json keys through a streaming script to replace '.' for '_'. *Example:* {{json = {"a":{"b": 1}, "c.d": 2}}} {{get_json_object(json, "a.b") returns: 1}} {{get_json_object(json, "c.d") returns: NULL}} In the present implementation of get_json_object, c.d is not addressable. *Proposal:* The desired behavior would be to allow the JSON path to be escape-able, like so: {{get_json_object(json, '$.c\\.d') would return: 2}} Affects Version/s: (was: 0.8.1) Fix Version/s: (was: 0.9.0) > Allow escape character in get_json_object > - > > Key: HIVE-2927 > URL: https://issues.apache.org/jira/browse/HIVE-2927 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Sean McNamara > Original Estimate: 0h > Remaining Estimate: 0h > > *Background:* > get_json_object extracts json objects from a json string based on a specified > path. > *Problem:* > The current implementation of get_json_object can't see keys with a '.' in > them. Our data contains '.' in the keys, so we have to filter our json keys > through a streaming script to replace '.' for '_'. > *Example:* > {{json = {"a":{"b": 1}, "c.d": 2}}} > {{get_json_object(json, "a.b") returns: 1}} > {{get_json_object(json, "c.d") returns: NULL}} > In the present implementation of get_json_object, c.d is not addressable. > *Proposal:* > The desired behavior would be to allow the JSON path to be escape-able, like > so: > {{get_json_object(json, '$.c.d') would return: 2}} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2927) Allow escape character in get_json_object
Allow escape character in get_json_object - Key: HIVE-2927 URL: https://issues.apache.org/jira/browse/HIVE-2927 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Affects Versions: 0.8.1 Reporter: Sean McNamara Fix For: 0.9.0 *Background:* get_json_object extracts json objects from a json string based on a specified path. *Problem:* The current implementation of get_json_object can't see keys with a '.' in them. Our data contains '.' in the keys, so we have to filter our json keys through a streaming script to replace '.' for '_'. *Example:* {{json = {"a":{"b": 1}, "c.d": 2}}} {{get_json_object(json, "a.b") returns: 1}} {{get_json_object(json, "c.d") returns: NULL}} In the present implementation of get_json_object, c.d is not addressable. *Proposal:* The desired behavior would be to allow the JSON path to be escape-able, like so: {{get_json_object(json, '$.c\\.d') would return: 2}} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Attachment: HIVE-2923.patch The problem is that the ZooKeeper client is used before it is connected (may or may not occur depending on environment). I could reproduce this yesterday on my Mac but not this morning. Attached patch will add a wait for SyncConnected with timeout to address this. Can you please try the patch? > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > Fix For: 0.9.0 > > Attachments: HIVE-2923.patch > > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Fix Version/s: 0.9.0 Affects Version/s: 0.9.0 Status: Patch Available (was: Open) > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > Fix For: 0.9.0 > > Attachments: HIVE-2923.patch > > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2504) Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory
[ https://issues.apache.org/jira/browse/HIVE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated HIVE-2504: - Hadoop Flags: (was: Reviewed) Status: Patch Available (was: Reopened) Unit tests pass with ant clean package test > Warehouse table subdirectories should inherit the group permissions of the > warehouse parent directory > - > > Key: HIVE-2504 > URL: https://issues.apache.org/jira/browse/HIVE-2504 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach >Assignee: Rohini Palaniswamy > Fix For: 0.9.0 > > Attachments: HIVE-2504.patch, HIVE-2504.patch > > > When the Hive Metastore creates a subdirectory in the Hive warehouse for > a new table it does so with the default HDFS permissions. Since the default > dfs.umask value is 022, this means that the new subdirectory will not inherit > the > group write permissions of the hive warehouse directory. > We should make the umask used by Warehouse.mkdirs() configurable, and set > it to use a default value of 002. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2504) Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory
[ https://issues.apache.org/jira/browse/HIVE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated HIVE-2504: - Attachment: HIVE-2504.patch Made the newly created subdirectories get the same permissions as the parent by setting it explicitly on them. The table directories will get the permissions of the database directory and the partition directories will get the permissions of the table directory. Removed the hive.files.umask.value configuration parameter. > Warehouse table subdirectories should inherit the group permissions of the > warehouse parent directory > - > > Key: HIVE-2504 > URL: https://issues.apache.org/jira/browse/HIVE-2504 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach >Assignee: Rohini Palaniswamy > Fix For: 0.9.0 > > Attachments: HIVE-2504.patch, HIVE-2504.patch > > > When the Hive Metastore creates a subdirectory in the Hive warehouse for > a new table it does so with the default HDFS permissions. Since the default > dfs.umask value is 022, this means that the new subdirectory will not inherit > the > group write permissions of the hive warehouse directory. > We should make the umask used by Warehouse.mkdirs() configurable, and set > it to use a default value of 002. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned HIVE-2923: -- Assignee: Thomas Weise (was: Ashutosh Chauhan) > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2917) Add support for various charsets in LazySimpleSerDe
[ https://issues.apache.org/jira/browse/HIVE-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2917: -- Attachment: HIVE-2917.D2619.1.patch flyinggarden requested code review of "HIVE-2917 [jira] Add support for various charsets in LazySimpleSerDe". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2917 HIVE-2917: Add support for various charsets in LazySimpleSerDe Currently hive can only serialize/deserialize data encoded in utf-8. It would be useful to specify the data's charset when creating the table. The idea is to add a new keyword CHARSET to set charset at table level. For example: CREATE TABLE tbl1 (col1 STRING) ROW FORMAT CHARET "GBK" DELIMITED FIELDS TERMINATED BY '\t'; Another place to use CHARSET is in TRANSFORM clause. For example: SELECT TRANSFORM(col1, col2) ROW FORMAT CHARSET 'gbk' USING 'some_script' AS (col3, col4) ROW FORMAT CHARSET 'utf-8'; TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2619 AFFECTED FILES hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java data/files/gbk.txt serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyArrayMapStruct.java serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyCharset.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyString.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyUnionObjectInspector.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyStringObjectInspector.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyListObjectInspector.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyMapObjectInspector.java serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py serde/src/gen/thrift/gen-cpp/serde_constants.cpp serde/src/gen/thrift/gen-cpp/serde_constants.h serde/src/gen/thrift/gen-rb/serde_constants.rb serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java serde/src/gen/thrift/gen-php/serde/serde_constants.php serde/if/serde.thrift ql/src/test/results/clientpositive/charset.q.out ql/src/test/results/clientpositive/input35.q.out ql/src/test/results/clientpositive/input36.q.out ql/src/test/results/clientpositive/transform_charset.q.out ql/src/test/queries/clientpositive/transform_charset.q ql/src/test/queries/clientpositive/charset.q ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6027/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Add support for various charsets in LazySimpleSerDe > --- > > Key: HIVE-2917 > URL: https://issues.apache.org/jira/browse/HIVE-2917 > Project: Hive > Issue Type: New Feature > Components: CLI, Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Kai Zhang > Attachments: HIVE-2917.1.patch.txt, HIVE-2917.2.patch.txt, > HIVE-2917.3.patch.txt, HIVE-2917.D2619.1.patch > > > Currently hive can only serialize/deserialize data encoded in utf-8. > It would be useful to specify the data's charset when creating the table. > The idea is to add a new keyword CHARSET to set charset at table level. > For example: > CREATE TABLE tbl1 (col1 STRING) ROW FORMAT CHARET "GBK" DELIMITED FIELDS > TERMINATED BY '\t'; > Another place to use CHARSET is in TRANSFORM clause. > For example: > SELECT TRANSFORM(col1, col2) ROW FORMAT CHARSET 'gbk' > USING 'some_script' > AS (col3, col4) ROW FORMAT CHARSET 'utf-8'; -- This message is automatically generated by JIRA. If you think it was sent incorrectly,
[jira] [Commented] (HIVE-2822) Add JSON output to the hive ddl commands
[ https://issues.apache.org/jira/browse/HIVE-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247079#comment-13247079 ] Hudson commented on HIVE-2822: -- Integrated in Hive-trunk-h0.21 #1353 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1353/]) HIVE-2822: Second part of it (ctdean via Ashutosh Chauhan) (Revision 1309667) HIVE-2822 [jira] Add JSON output to the hive ddl commands (Chris Dean via Ashutosh Chauhan) Summary: JSON output for DDL commands By setting the variable hive.format=json then ddl commands will produce json output suitable for client side parsing. For example, to list all the databases one might get: { "databases": [ "default" ] } The goal is to have an option to produce JSON output of the DDL commands that is easily machine parseable. For example, "desc my_table" currently gives idbigint user string and we want to allow a json output: { "columns": [ {"name": "id", "type": "bigint"}, {"name": "user", "type": "string"} ] } Test Plan: Run the provided unit tests Reviewers: JIRA, ashutoshc Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D2475 (Revision 1309666) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309667 Files : * /hive/trunk/ql/src/test/results/clientnegative/column_rename1.q.out * /hive/trunk/ql/src/test/results/clientnegative/column_rename4.q.out hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309666 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MapBuilder.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatter.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java * /hive/trunk/ql/src/test/queries/clientpositive/describe_database_json.q * /hive/trunk/ql/src/test/queries/clientpositive/describe_table_json.q * /hive/trunk/ql/src/test/queries/clientpositive/misc_json.q * /hive/trunk/ql/src/test/queries/clientpositive/partitions_json.q * /hive/trunk/ql/src/test/results/clientnegative/database_create_already_exists.q.out * /hive/trunk/ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out * /hive/trunk/ql/src/test/results/clientpositive/describe_database_json.q.out * /hive/trunk/ql/src/test/results/clientpositive/describe_table_json.q.out * /hive/trunk/ql/src/test/results/clientpositive/misc_json.q.out * /hive/trunk/ql/src/test/results/clientpositive/partitions_json.q.out > Add JSON output to the hive ddl commands > > > Key: HIVE-2822 > URL: https://issues.apache.org/jira/browse/HIVE-2822 > Project: Hive > Issue Type: Improvement >Reporter: Chris Dean >Assignee: Chris Dean > Fix For: 0.9.0 > > Attachments: HIVE-2822.03-branch0-8.patch, HIVE-2822.03.patch, > HIVE-2822.03b.patch, HIVE-2822.04-branch-08.patch, > HIVE-2822.05-branch0-8-1.patch, HIVE-2822.05-branch0-8.patch, > HIVE-2822.05.patch, HIVE-2822.D2475.1.patch, HIVE-2822.D2475.2.patch, > HIVE-2822.D2475.3.patch, HIVE-2822.D2475.4.patch, > hive-json-01-branch0-8.patch, hive-json-01.patch, > hive-json-02-branch0-8.patch, hive-json-02.patch > > > The goal is to have an option to produce JSON output of the DDL commands that > is easily machine parseable. > For example, "desc my_table" currently gives > {noformat} > idbigint > user string > {noformat} > and we want to allow a json output: > {noformat} > { > "columns": [ > {"name": "id", "type": "bigint"}, > {"name": "user", "type": "string"} > ] > } > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247075#comment-13247075 ] Ashutosh Chauhan commented on HIVE-2764: Those tests passed. So, thats a progress. But following new ones failed. * rcfile_merge3.q * rcfile_createas1.q > Obtain delegation tokens for MR jobs in secure hbase setup > > > Key: HIVE-2764 > URL: https://issues.apache.org/jira/browse/HIVE-2764 > Project: Hive > Issue Type: Improvement > Components: HBase Handler, Security >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, > HIVE-2764.D2205.3.patch, HIVE-2764.D2205.4.patch, HIVE-2764_v0.patch > > > As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to > obtain delegation tokens for hbase and save it in jobconf, so that tasks can > access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2879) Add "rat" target to build to look for missing license headers
[ https://issues.apache.org/jira/browse/HIVE-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2879: --- Resolution: Fixed Fix Version/s: 0.9.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Owen! > Add "rat" target to build to look for missing license headers > - > > Key: HIVE-2879 > URL: https://issues.apache.org/jira/browse/HIVE-2879 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 0.9.0 > > Attachments: HIVE-2879.D2373.1.patch, HIVE-2879.D2373.2.patch, > HIVE-2879.D2373.3.patch > > > We should have a "rat" target that can generate an Apache RAT report. RAT is > used for finding missing license headers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2559) Add target to install Hive JARs/POMs in the local Maven cache
[ https://issues.apache.org/jira/browse/HIVE-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2559: --- Resolution: Fixed Fix Version/s: 0.9.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Alan! > Add target to install Hive JARs/POMs in the local Maven cache > - > > Key: HIVE-2559 > URL: https://issues.apache.org/jira/browse/HIVE-2559 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Affects Versions: 0.9.0 >Reporter: Alejandro Abdelnur >Assignee: Alan Gates >Priority: Critical > Fix For: 0.9.0 > > Attachments: HIVE-2559.patch > > > HIVE-2391 is producing usable Maven artifacts. > However, it only as a target to deploy/publish those artifacts to Apache > Maven repos. > There should be a new target to locally install Hive Maven artifacts, thus > enabling their use from other projects before they are committed/publish to > Apache Maven (this is critical to test patches that may address issues in > downstream components). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira