[jira] [Updated] (HIVE-2726) COALESCE(NULL, NULL) NPEs
[ https://issues.apache.org/jira/browse/HIVE-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2726: -- Attachment: HIVE-2726.D1317.2.patch jonchang updated the revision "HIVE-2726 [jira] COALESCE(NULL, NULL) NPEs". Reviewers: JIRA, jsichi, raghotham, njain Change the way it works so it no loner applies to ARRAY() and MAP(). Also clean up the way SORT_ARRAY() works. REVISION DETAIL https://reviews.facebook.net/D1317 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java ql/src/test/queries/clientpositive/udf_coalesce.q ql/src/test/results/clientpositive/udf_coalesce.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out > COALESCE(NULL, NULL) NPEs > - > > Key: HIVE-2726 > URL: https://issues.apache.org/jira/browse/HIVE-2726 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Chang >Assignee: Jonathan Chang >Priority: Minor > Attachments: HIVE-2726.D1317.1.patch, HIVE-2726.D1317.1.patch, > HIVE-2726.D1317.2.patch, HIVE-2726.D1677.1.patch > > > ReturnOIResolver returns null when it gets no non-void arguments. This is > bad. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2726) COALESCE(NULL, NULL) NPEs
[ https://issues.apache.org/jira/browse/HIVE-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2726: -- Attachment: HIVE-2726.D1677.1.patch jonchang requested code review of "HIVE-2726 [jira] COALESCE(NULL, NULL) NPEs". Reviewers: JIRA HIVE-2726 ReturnOIResolver returns null when it gets no non-void arguments. This is bad. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1677 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java ql/src/test/queries/clientpositive/udf_coalesce.q ql/src/test/results/clientpositive/udf_coalesce.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3567/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > COALESCE(NULL, NULL) NPEs > - > > Key: HIVE-2726 > URL: https://issues.apache.org/jira/browse/HIVE-2726 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Chang >Assignee: Jonathan Chang >Priority: Minor > Attachments: HIVE-2726.D1317.1.patch, HIVE-2726.D1317.1.patch, > HIVE-2726.D1677.1.patch > > > ReturnOIResolver returns null when it gets no non-void arguments. This is > bad. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2794) Aggregations without grouping should return NULL when applied to partitioning column of a partitionless table
[
https://issues.apache.org/jira/browse/HIVE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204295#comment-13204295
]
Carl Steinbach commented on HIVE-2794:
--
"SELECT MAX(p) FROM empty" should return NULL, but instead it returns an empty
result set:
{noformat}
hive> CREATE TABLE empty (c INT) PARTITIONED BY (p INT);
hive> SELECT MAX(c) FROM empty;
NULL
hive> SELECT MAX(p) FROM empty;
hive> ALTER TABLE empty ADD PARTITION (p=1);
hive> SELECT MAX(p) FROM empty;
1
hive>
{noformat}
> Aggregations without grouping should return NULL when applied to partitioning
> column of a partitionless table
> -
>
> Key: HIVE-2794
> URL: https://issues.apache.org/jira/browse/HIVE-2794
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Carl Steinbach
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2794) Aggregations without grouping should return NULL when applied to partitioning column of a partitionless table
Aggregations without grouping should return NULL when applied to partitioning column of a partitionless table - Key: HIVE-2794 URL: https://issues.apache.org/jira/browse/HIVE-2794 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2749) CONV returns incorrect results sometimes
[ https://issues.apache.org/jira/browse/HIVE-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2749: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed. Thanks Jonathan > CONV returns incorrect results sometimes > > > Key: HIVE-2749 > URL: https://issues.apache.org/jira/browse/HIVE-2749 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Chang >Assignee: Jonathan Chang > Attachments: HIVE-2749.D1437.1.patch, HIVE-2749.D1437.2.patch, > HIVE-2749.D1437.2.patch > > > ...because it fails to reset state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Showing wrong count after importing table in Hive
Hello All, I have imported near about 10 tables in Hive from MS SQL Server. But when I try to cross check the records in Hive in one of the Table I have found more record when I run the query (select count(*) from tblName;). Then I have drop the that Table and again imported it in Hive. I have observed in Console Logs that (Retrieved 203 records). And then I tried again for (select count(*) from tblName;) and I got the count as 298. I dont understand this why this happens. Is anything is wrong in query or it happens due to some incorrect command of sqoop-import. All other table records are fine. I got stuck here and I had spend much time to search for this. Pls help me out from this. -- Thanks and Regards, Bhavesh Shah
[jira] [Updated] (HIVE-2793) Disable loadpart_err.q on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2793: - Attachment: HIVE-2793.D1665.1.patch > Disable loadpart_err.q on 0.23 > -- > > Key: HIVE-2793 > URL: https://issues.apache.org/jira/browse/HIVE-2793 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2793.D1665.1.patch, HIVE-2793.D1665.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2793) Disable loadpart_err.q on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2793: -- Attachment: HIVE-2793.D1665.1.patch cwsteinbach requested code review of "HIVE-2793 [jira] Disable loadpart_err.q on 0.23". Reviewers: JIRA HIVE-2793. Add 0.23 to list of excluded Hadoop versions for loadpart_err.q TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1665 AFFECTED FILES ql/src/test/queries/clientpositive/loadpart_err.q MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3555/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Disable loadpart_err.q on 0.23 > -- > > Key: HIVE-2793 > URL: https://issues.apache.org/jira/browse/HIVE-2793 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2793.D1665.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2793) Disable loadpart_err.q on 0.23
Disable loadpart_err.q on 0.23 -- Key: HIVE-2793 URL: https://issues.apache.org/jira/browse/HIVE-2793 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2793) Disable loadpart_err.q on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204224#comment-13204224 ] Carl Steinbach commented on HIVE-2793: -- This testcase was previously disabled for all Hadoop versions in HIVE-1491. On 0.23 this test fails with the same error as the one produced on 0.20 (if enabled). > Disable loadpart_err.q on 0.23 > -- > > Key: HIVE-2793 > URL: https://issues.apache.org/jira/browse/HIVE-2793 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: optimization of tree plan generated by Hive...
Hi Alexis, Work is already underway to add the YSmart optimizer to Hive. Please take a look at https://issues.apache.org/jira/browse/HIVE-2206. Thanks. Carl On Wed, Feb 8, 2012 at 6:17 PM, Alexis De La Cruz Toledo < alexis...@gmail.com> wrote: > Hi! My name is Alexis. I am a master student of Cinvestav, DF, México. > Actually I am doing my thesis work and I would like to participate in > Google Summer of Code 2012 ( > http://google-melange.appspot.com/gsoc/events/google/gsoc2012) > I'm interesting in improve Hive and I have been studying hadoop and hive. > > I have interesting about the tree plan generated by Hive. > Call me the attention that Hive read many times the same table > and generate many jobs hadoop when the query can be > expressed in less Jobs and with only one read of the table > if I program the same query in hadoop. > > I think that I can reduce the number of jobs to process a query > and read the tables one time too, no matter if used it on several jobs. > > The solution could be raised of two ways: > > 1. Changing the part when the DAG is created, making the optimizations in > this moment. > 2. After that the DAG is created, we can apply the optimizations, this > optimizations can be implemented in another class. > > Where could I do this? I think that the method that compile the queries is > the method compile in class Driver, am I right? > Can someone guide me where I could implement it? > > There is a paper which discussed what I say > http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-11-7.pdf > We can take it and improve or implement us own ideas. > > Personally I would like to do the second options due to time. > > By another hand, Someone is interested to work with me and be my mentor in > Google Summer Code 2012? > > Thanks. > > Regards. > > -- > Ing. Alexis de la Cruz Toledo. > *Av. Instituto Politécnico Nacional No. 2508 Col. San Pedro Zacatenco. > México, > D.F, 07360 * > *CINVESTAV, DF.* >
[jira] [Commented] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204202#comment-13204202 ] Carl Steinbach commented on HIVE-2782: -- @Ashutosh: I also found what I think is a bug in the way SUBSTR handles BINARY inputs. Can you please take a look? https://issues.apache.org/jira/browse/HIVE-2792 > New BINARY type produces unexpected results with supported UDFS when using > MapReduce2 > - > > Key: HIVE-2782 > URL: https://issues.apache.org/jira/browse/HIVE-2782 > Project: Hive > Issue Type: Bug >Reporter: Zhenxiao Luo >Assignee: Carl Steinbach > Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch > > > When using MapReduce2 for Hive > ba_table_udfs is failing with unexpected output: > [junit] Begin query: ba_table_udfs.q > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I > lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime > -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I > java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I > Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* > more -I job_[0-9]*_[0-9]* -I USING 'java -cp > /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out > > /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out > [junit] 20,26c20,26 > [junit] < 2 10val_101 > [junit] < 3 164val_164 1 > [junit] < 3 150val_150 1 > [junit] < 2 18val_181 > [junit] < 3 177val_177 1 > [junit] < 2 12val_121 > [junit] < 2 11val_111 > [junit] — > [junit] > 3 120val_120 1 > [junit] > 3 192val_192 1 > [junit] > 3 119val_119 1 > [junit] > 3 187val_187 1 > [junit] > 3 176val_176 1 > [junit] > 3 199val_199 1 > [junit] > 3 118val_118 1 > [junit] Exception: Client execution results failed with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] junit.framework.AssertionFailedError: Client execution results failed > with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] at junit.framework.Assert.fail(Assert.java:50) > [junit] at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129) > [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [junit] at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > [junit] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [junit] at java.lang.reflect.Method.invoke(Method.java:616) > [junit] at junit.framework.TestCase.runTest(TestCase.java:168) > [junit] at junit.framework.TestCase.runBare(TestCase.java:134) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) > [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) > [junit] at junit.framework.TestResult.run(TestResult.java:113) > [junit] at junit.framework.TestCase.run(TestCase.java:124) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) > [junit] at junit.framework.TestSuite.run(TestSuite.java:238) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs.) > [junit] Cleaning up TestCliDriver > [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.751 sec > [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED > [for] /home/cloudera/Code/hive/ql/build.xml: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build.xml:328: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build-common.xml:453: Tests failed! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2792) SUBSTR(CAST( AS BINARY)) produces unexpected results
[
https://issues.apache.org/jira/browse/HIVE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204200#comment-13204200
]
Carl Steinbach commented on HIVE-2792:
--
{noformat}
SELECT
value,
SUBSTR(value, 4, 5),
SUBSTR(CAST(value AS BINARY), 4, 5)
FROM src
ORDER BY value;
...
val_0 _0 039<-- Third column of the first three rows should be the
same, right?
val_0 _0 096
val_0 _0 073
val_10 _10 105
val_100 _100100
val_100 _100100
val_103 _103103
val_103 _103103
val_104 _104104
val_104 _104104
val_105 _105105
val_11 _11 116
val_111 _111111
val_113 _113113
val_113 _113113
val_114 _114114
val_116 _116116
val_118 _118118
val_118 _118118
val_119 _119119
val_119 _119119
val_119 _119119
val_12 _12 124
val_12 _12 124
...
{noformat}
> SUBSTR(CAST( AS BINARY)) produces unexpected results
>
>
> Key: HIVE-2792
> URL: https://issues.apache.org/jira/browse/HIVE-2792
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Carl Steinbach
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2792) SUBSTR(CAST( AS BINARY)) produces unexpected results
SUBSTR(CAST( AS BINARY)) produces unexpected results Key: HIVE-2792 URL: https://issues.apache.org/jira/browse/HIVE-2792 Project: Hive Issue Type: Bug Components: UDF Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1249 - Still Failing
Changes for Build #1207 [namit] HIVE-2718 NPE in union followed by join (He Yongqiang via namit) Changes for Build #1208 Changes for Build #1209 Changes for Build #1210 [namit] HIVE-2674 get_partitions_ps throws TApplicationException if table doesn't exist (Kevin Wilfong via namit) Changes for Build #1211 [cws] HIVE-2203. Extend concat_ws() UDF to support arrays of strings (Zhenxiao Luo via cws) [cws] HIVE-2279. Implement sort(array) UDF (Zhenxiao Luo via cws) Changes for Build #1212 [hashutosh] HIVE-2589 : Newly created partition should inherit properties from table (Ashutosh Chauhan) Changes for Build #1213 Changes for Build #1214 Changes for Build #1215 Changes for Build #1216 [cws] HIVE-2724. Remove unused lib/log4j-1.2.15.jar (Andrew Bayer via cws) Changes for Build #1217 Changes for Build #1218 Changes for Build #1219 [hashutosh] HIVE-2665 : Support for metastore service specific HADOOP_OPTS environment setting (thw via hashutosh) Changes for Build #1220 [namit] HIVE-2727 add a testcase for partitioned view on union and base tables have index (He Yongqiang via namit) Changes for Build #1221 [hashutosh] HIVE-2746 : Metastore client doesn't log properly in case of connection failure to server (hashutosh) [cws] HIVE-2698 [jira] Enable Hadoop-1.0.0 in Hive (Enis Söztutar via Carl Steinbach) Summary: third version of the patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach CC: cwsteinbach, enis Differential Revision: https://reviews.facebook.net/D1389 Changes for Build #1222 [namit] HIVE-2750 Hive multi group by single reducer optimization causes invalid column reference error (Kevin Wilfong via namit) Changes for Build #1223 Changes for Build #1224 [cws] HIVE-2734 [jira] Fix some nondeterministic test output (Zhenxiao Luo via Carl Steinbach) Summary: HIVE-2734: Fix some nondeterministic test output Many Hive query tests lack an ORDER BY clause, and consequently the ordering of the rows in the result set is nondeterministic: groupby1_limit input11_limit input1_limit input_lazyserde join18_multi_distinct join_1to1 join_casesensitive join_filters join_nulls merge3 rcfile_columnar rcfile_lazydecompress rcfile_union sample10 udf_sentences union24 columnarserde_create_shortcut combine1 global_limit Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach CC: zhenxiao, cwsteinbach Differential Revision: https://reviews.facebook.net/D1449 [namit] HIVE-2754 NPE in union with lateral view (Yongqiang He via namit) Changes for Build #1225 Changes for Build #1226 Changes for Build #1227 [namit] HIVE-2755 union follwowed by union_subq does not work if the subquery union has reducers (He Yongqiang via namit) Changes for Build #1228 Changes for Build #1229 [hashutosh] HIVE-2735: PlanUtils.configureTableJobPropertiesForStorageHandler() is not called for partitioned table (sushanth via ashutosh) Changes for Build #1230 [cws] HIVE-2760 [jira] TestCliDriver should log elapsed time Summary: HIVE-2760. TestCliDriver should log elapsed time Test Plan: EMPTY Reviewers: JIRA, ashutoshc Reviewed By: ashutoshc CC: ashutoshc, cwsteinbach Differential Revision: https://reviews.facebook.net/D1503 [cws] HIVE-2662 [jira] Add Ant configuration property for dumping classpath of tests Summary: HIVE-2662. Add Ant configuration property for dumping classpath of tests Test Plan: EMPTY Reviewers: JIRA, jsichi, ashutoshc Reviewed By: ashutoshc CC: ashutoshc Differential Revision: https://reviews.facebook.net/D903 Changes for Build #1231 [hashutosh] HIVE_2645: Hive Web Server startup messages logs incorrect path it is searching for WAR (Edward Capriolo via Ashutosh Chauhan) Changes for Build #1232 Changes for Build #1233 [sdong] HIVE-2249 When creating constant expression for numbers, try to infer type from another comparison operand, instead of trying to use integer first, and then long and double (Zhiqiu Kong via Siying Dong) Changes for Build #1234 Changes for Build #1235 [heyongqiang] HIVE-2765 hbase handler uses ZooKeeperConnectionException which is not compatible with HBase versions other than 0.89 (Pei Yue via He Yongqiang) Changes for Build #1236 Changes for Build #1237 Changes for Build #1238 [heyongqiang] HIVE-2772 [jira] make union31.q deterministic (Namit Jain via Yongqiang He) Summary: https://issues.apache.org/jira/browse/HIVE-2772 HIVE-2772 Test Plan: EMPTY Reviewers: JIRA, ashutoshc Reviewed By: ashutoshc CC: ashutoshc Differential Revision: https://reviews.facebook.net/D1557 [kevinwilfong] HIVE-2758 Metastore is caching too aggressively (Kevin Wilfong reviewed by Carl Steinbach) Changes for Build #1239 Changes for Build #1240 [namit] HIVE-2762 Alter Table Partition Concatenate Fails On Certain Characters (Kevin Wilfong via namit) Changes for Build #1241 [namit] HIVE-2756 Views should be added to th
[jira] [Assigned] (HIVE-2791) filter is still removed due to regression of HIVE-1538 althougth HIVE-2344
[ https://issues.apache.org/jira/browse/HIVE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-2791: -- Assignee: binlijin Binlijin, Is it ready for review? Mark Patch Available in that case. Also, reupload the patch granting the ASF perms. > filter is still removed due to regression of HIVE-1538 althougth HIVE-2344 > -- > > Key: HIVE-2791 > URL: https://issues.apache.org/jira/browse/HIVE-2791 > Project: Hive > Issue Type: Bug >Reporter: binlijin >Assignee: binlijin > Attachments: HIVE-2791.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2759) Change global_limit.q into linux format file
[ https://issues.apache.org/jira/browse/HIVE-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204174#comment-13204174 ] Hudson commented on HIVE-2759: -- Integrated in Hive-trunk-h0.21 #1249 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1249/]) HIVE-2759 Change global_limit.q into linux format file (Zhenxiao Luo via namit) namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242071 Files : * /hive/trunk/ql/src/test/queries/clientpositive/global_limit.q > Change global_limit.q into linux format file > > > Key: HIVE-2759 > URL: https://issues.apache.org/jira/browse/HIVE-2759 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Zhenxiao Luo >Assignee: Zhenxiao Luo > Attachments: HIVE-2759.D1497.1.patch, HIVE-2759.D1497.1.patch > > > global_limit.q is in dos format, and has "\ No newline at end of file". > Needs to be changed into linux format, removing all "^M" at the end of each > line. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Thrift build
Thanks Namit. $ svn co svn+ssh://tubbs/svnhive/hadoop/external/thrift-0.7.0 svn: To better debug SSH connection problems, remove the -q option from 'ssh' in the [tunnels] section of your Subversion configuration file. svn: Network connection closed unexpectedly I am not able to reach it. Am I missing anything? Thanks, Aniket On Wed, Feb 8, 2012 at 3:58 PM, Namit Jain wrote: > Check out svn+ssh://tubbs/svnhive/hadoop/external/thrift-0.7.0 > > And set your thrift.home to the above > > > On 2/8/12 12:32 PM, "Aniket Mokashi" wrote: > > >Hi, > > > >I made some changes to hive_metastore.thrift to add a few new apis. I am > >trying to generate code with thrift, but I am hitting following errors- > > > >lgmac-amokashi:metastore amokashi$ ant thriftif -Dthrift.home=/usr/local > >Buildfile: /work/projects/hive-trunk/metastore/build.xml > > > >check-thrift-home: > > [echo] Project: metastore > > > >thriftif: > > [echo] Project: metastore > > [echo] Executing /usr/local/bin/thrift on > >metastore/if/hive_metastore.thrift > > [exec] > >[ERROR:/work/projects/hive-trunk/metastore/if/hive_metastore.thrift:224] > >(last token was 'fb303.FacebookService') > > [exec] Service > >"fb303[WARNING:/work/projects/hive-trunk/metastore/if/hive_metastore.thrif > >t:6] > >Could not find include file share/fb303/if/fb303.thrift.FacebookService" > >has not been defined. > > [exec] > > [exec] > > > >BUILD FAILED > >/work/projects/hive-trunk/metastore/build.xml:29: exec returned: 1 > > > >Please let me know if I am missing anything. > > > >Thanks, > >Aniket > > -- "...:::Aniket:::... Quetzalco@tl"
Re: Thrift build
Check out svn+ssh://tubbs/svnhive/hadoop/external/thrift-0.7.0 And set your thrift.home to the above On 2/8/12 12:32 PM, "Aniket Mokashi" wrote: >Hi, > >I made some changes to hive_metastore.thrift to add a few new apis. I am >trying to generate code with thrift, but I am hitting following errors- > >lgmac-amokashi:metastore amokashi$ ant thriftif -Dthrift.home=/usr/local >Buildfile: /work/projects/hive-trunk/metastore/build.xml > >check-thrift-home: > [echo] Project: metastore > >thriftif: > [echo] Project: metastore > [echo] Executing /usr/local/bin/thrift on >metastore/if/hive_metastore.thrift > [exec] >[ERROR:/work/projects/hive-trunk/metastore/if/hive_metastore.thrift:224] >(last token was 'fb303.FacebookService') > [exec] Service >"fb303[WARNING:/work/projects/hive-trunk/metastore/if/hive_metastore.thrif >t:6] >Could not find include file share/fb303/if/fb303.thrift.FacebookService" >has not been defined. > [exec] > [exec] > >BUILD FAILED >/work/projects/hive-trunk/metastore/build.xml:29: exec returned: 1 > >Please let me know if I am missing anything. > >Thanks, >Aniket
[jira] [Commented] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204109#comment-13204109 ] Zhenxiao Luo commented on HIVE-2782: @carl: No problem with the original patch. @Ashutosh: My understanding is, The original query: select length (cast(src.key as binary)) as len, concat(cast(src.key as binary), cast(src.value as binary)), substr(src.value, 5,1) as sub from src order by sub limit 10; Has the order by on sub, which is the fifth character of src.value. In the result set, we could see sub is ordered(only showing sub == 1) While, for the len and concat with the same sub, their orders are not deterministic. Eg. 2 10val_101 3 120val_120 1 both has sub == 1. but the order of these two tuples is not deterministic. since we only show the top 10 results(limit 10), and there seems are more than 7 tuples with sub == 1, so what shows in the result set is non-deterministic. Carl solves the problem by providing order into the query. Correct me if I am wrong. Thanks > New BINARY type produces unexpected results with supported UDFS when using > MapReduce2 > - > > Key: HIVE-2782 > URL: https://issues.apache.org/jira/browse/HIVE-2782 > Project: Hive > Issue Type: Bug >Reporter: Zhenxiao Luo >Assignee: Carl Steinbach > Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch > > > When using MapReduce2 for Hive > ba_table_udfs is failing with unexpected output: > [junit] Begin query: ba_table_udfs.q > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I > lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime > -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I > java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I > Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* > more -I job_[0-9]*_[0-9]* -I USING 'java -cp > /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out > > /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out > [junit] 20,26c20,26 > [junit] < 2 10val_101 > [junit] < 3 164val_164 1 > [junit] < 3 150val_150 1 > [junit] < 2 18val_181 > [junit] < 3 177val_177 1 > [junit] < 2 12val_121 > [junit] < 2 11val_111 > [junit] — > [junit] > 3 120val_120 1 > [junit] > 3 192val_192 1 > [junit] > 3 119val_119 1 > [junit] > 3 187val_187 1 > [junit] > 3 176val_176 1 > [junit] > 3 199val_199 1 > [junit] > 3 118val_118 1 > [junit] Exception: Client execution results failed with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] junit.framework.AssertionFailedError: Client execution results failed > with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] at junit.framework.Assert.fail(Assert.java:50) > [junit] at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129) > [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [junit] at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > [junit] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [junit] at java.lang.reflect.Method.invoke(Method.java:616) > [junit] at junit.framework.TestCase.runTest(TestCase.java:168) > [junit] at junit.framework.TestCase.runBare(TestCase.java:134) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) > [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) > [junit] at junit.framework.TestResult.run(TestResult.java:113) > [junit] at junit.framework.TestCase.run(TestCase.java:124) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) > [junit] at junit.framework.TestSuite.run(TestSuite.java:238) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs.) > [junit] Cleaning up TestCliDri
[jira] [Commented] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[
https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204094#comment-13204094
]
Ashutosh Chauhan commented on HIVE-2782:
{noformat}
[junit] < 2 10val_101
[junit] < 3 164val_164 1
[junit] < 3 150val_150 1
[junit] < 2 18val_181
[junit] < 3 177val_177 1
[junit] < 2 12val_121
[junit] < 2 11val_111
[junit] —
[junit] > 3 120val_120 1
[junit] > 3 192val_192 1
[junit] > 3 119val_119 1
[junit] > 3 187val_187 1
[junit] > 3 176val_176 1
[junit] > 3 199val_199 1
[junit] > 3 118val_118 1
{noformat}
If I am reading this correctly, this diff is not showing ordering issues of
rows, rather few of the rows are different then whats expected, which then is a
correctness issue. Isn't it ?
> New BINARY type produces unexpected results with supported UDFS when using
> MapReduce2
> -
>
> Key: HIVE-2782
> URL: https://issues.apache.org/jira/browse/HIVE-2782
> Project: Hive
> Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Carl Steinbach
> Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch
>
>
> When using MapReduce2 for Hive
> ba_table_udfs is failing with unexpected output:
> [junit] Begin query: ba_table_udfs.q
> [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is
> deprecated. Instead, use mapreduce.jobtracker.system.dir
> [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is
> deprecated. Instead, use mapreduce.cluster.local.dir
> [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I
> lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime
> -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I
> java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I
> Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]*
> more -I job_[0-9]*_[0-9]* -I USING 'java -cp
> /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out
>
> /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out
> [junit] 20,26c20,26
> [junit] < 2 10val_101
> [junit] < 3 164val_164 1
> [junit] < 3 150val_150 1
> [junit] < 2 18val_181
> [junit] < 3 177val_177 1
> [junit] < 2 12val_121
> [junit] < 2 11val_111
> [junit] —
> [junit] > 3 120val_120 1
> [junit] > 3 192val_192 1
> [junit] > 3 119val_119 1
> [junit] > 3 187val_187 1
> [junit] > 3 176val_176 1
> [junit] > 3 199val_199 1
> [junit] > 3 118val_118 1
> [junit] Exception: Client execution results failed with error code = 1
> [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false"
> to get more logs.
> [junit] junit.framework.AssertionFailedError: Client execution results failed
> with error code = 1
> [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false"
> to get more logs.
> [junit] at junit.framework.Assert.fail(Assert.java:50)
> [junit] at
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129)
> [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit] at java.lang.reflect.Method.invoke(Method.java:616)
> [junit] at junit.framework.TestCase.runTest(TestCase.java:168)
> [junit] at junit.framework.TestCase.runBare(TestCase.java:134)
> [junit] at junit.framework.TestResult$1.protect(TestResult.java:110)
> [junit] at junit.framework.TestResult.runProtected(TestResult.java:128)
> [junit] at junit.framework.TestResult.run(TestResult.java:113)
> [junit] at junit.framework.TestCase.run(TestCase.java:124)
> [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243)
> [junit] at junit.framework.TestSuite.run(TestSuite.java:238)
> [junit] at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
> [junit] at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
> [junit] at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
> [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false"
> to get more logs.)
> [junit] Cleaning up TestCliDriver
> [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.751 sec
> [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED
> [for] /home/cloudera/Code/hive/ql/bui
[jira] [Commented] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204092#comment-13204092 ] Carl Steinbach commented on HIVE-2782: -- @Zhenxiao: Can you please explain how this is an improvement on the version I provided in my patch? > New BINARY type produces unexpected results with supported UDFS when using > MapReduce2 > - > > Key: HIVE-2782 > URL: https://issues.apache.org/jira/browse/HIVE-2782 > Project: Hive > Issue Type: Bug >Reporter: Zhenxiao Luo >Assignee: Carl Steinbach > Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch > > > When using MapReduce2 for Hive > ba_table_udfs is failing with unexpected output: > [junit] Begin query: ba_table_udfs.q > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I > lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime > -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I > java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I > Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* > more -I job_[0-9]*_[0-9]* -I USING 'java -cp > /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out > > /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out > [junit] 20,26c20,26 > [junit] < 2 10val_101 > [junit] < 3 164val_164 1 > [junit] < 3 150val_150 1 > [junit] < 2 18val_181 > [junit] < 3 177val_177 1 > [junit] < 2 12val_121 > [junit] < 2 11val_111 > [junit] — > [junit] > 3 120val_120 1 > [junit] > 3 192val_192 1 > [junit] > 3 119val_119 1 > [junit] > 3 187val_187 1 > [junit] > 3 176val_176 1 > [junit] > 3 199val_199 1 > [junit] > 3 118val_118 1 > [junit] Exception: Client execution results failed with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] junit.framework.AssertionFailedError: Client execution results failed > with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] at junit.framework.Assert.fail(Assert.java:50) > [junit] at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129) > [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [junit] at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > [junit] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [junit] at java.lang.reflect.Method.invoke(Method.java:616) > [junit] at junit.framework.TestCase.runTest(TestCase.java:168) > [junit] at junit.framework.TestCase.runBare(TestCase.java:134) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) > [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) > [junit] at junit.framework.TestResult.run(TestResult.java:113) > [junit] at junit.framework.TestCase.run(TestCase.java:124) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) > [junit] at junit.framework.TestSuite.run(TestSuite.java:238) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs.) > [junit] Cleaning up TestCliDriver > [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.751 sec > [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED > [for] /home/cloudera/Code/hive/ql/build.xml: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build.xml:328: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build-common.xml:453: Tests failed! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204081#comment-13204081 ] Zhenxiao Luo commented on HIVE-2782: I think Carl is correct. Order by the fifth character of the src.value field does not guarantee deterministic ordering. How about: SELECT key, value, LENGTH(CAST(src.key AS BINARY)) as key_length, LENGTH(CAST(src.value AS BINARY)) as value_length, CONCAT(CAST(src.key AS BINARY), CAST(src.value AS BINARY)) as key_value_concat, SUBSTR(CAST(src.value AS BINARY), 1, 4) as value_substr FROM src ORDER BY key ASC, value ASC, key_length ASC, value_length ASC, key_value_concat ASC, value_substr ASC LIMIT 100; This should guarantee the ordering. > New BINARY type produces unexpected results with supported UDFS when using > MapReduce2 > - > > Key: HIVE-2782 > URL: https://issues.apache.org/jira/browse/HIVE-2782 > Project: Hive > Issue Type: Bug >Reporter: Zhenxiao Luo >Assignee: Carl Steinbach > Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch > > > When using MapReduce2 for Hive > ba_table_udfs is failing with unexpected output: > [junit] Begin query: ba_table_udfs.q > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I > lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime > -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I > java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I > Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* > more -I job_[0-9]*_[0-9]* -I USING 'java -cp > /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out > > /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out > [junit] 20,26c20,26 > [junit] < 2 10val_101 > [junit] < 3 164val_164 1 > [junit] < 3 150val_150 1 > [junit] < 2 18val_181 > [junit] < 3 177val_177 1 > [junit] < 2 12val_121 > [junit] < 2 11val_111 > [junit] — > [junit] > 3 120val_120 1 > [junit] > 3 192val_192 1 > [junit] > 3 119val_119 1 > [junit] > 3 187val_187 1 > [junit] > 3 176val_176 1 > [junit] > 3 199val_199 1 > [junit] > 3 118val_118 1 > [junit] Exception: Client execution results failed with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] junit.framework.AssertionFailedError: Client execution results failed > with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] at junit.framework.Assert.fail(Assert.java:50) > [junit] at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129) > [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [junit] at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > [junit] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [junit] at java.lang.reflect.Method.invoke(Method.java:616) > [junit] at junit.framework.TestCase.runTest(TestCase.java:168) > [junit] at junit.framework.TestCase.runBare(TestCase.java:134) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) > [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) > [junit] at junit.framework.TestResult.run(TestResult.java:113) > [junit] at junit.framework.TestCase.run(TestCase.java:124) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) > [junit] at junit.framework.TestSuite.run(TestSuite.java:238) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs.) > [junit] Cleaning up TestCliDriver > [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.751 sec > [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED > [for] /home/cloudera/Code/hive/ql/build.xml: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build.xml:328: The following error occurred
[jira] [Commented] (HIVE-2372) java.io.IOException: error=7, Argument list too long
[
https://issues.apache.org/jira/browse/HIVE-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204037#comment-13204037
]
Luis Ramos commented on HIVE-2372:
--
As a hive user affected by this bug, I would have no problem with truncating
the variable to 20KB. That should be enough to give you an idea of the input
list. 2KB sounds low to me.
> java.io.IOException: error=7, Argument list too long
>
>
> Key: HIVE-2372
> URL: https://issues.apache.org/jira/browse/HIVE-2372
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Sergey Tryuber
>Priority: Critical
> Attachments: HIVE-2372.1.patch.txt
>
>
> I execute a huge query on a table with a lot of 2-level partitions. There is
> a perl reducer in my query. Maps worked ok, but every reducer fails with the
> following exception:
> 2011-08-11 04:58:29,865 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> Executing [/usr/bin/perl, , ]
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> tablename=null
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> partname=null
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> alias=null
> 2011-08-11 04:58:29,935 FATAL ExecReducer:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":129390185139228,"reducesinkkey1":"8AF163CA6F"},"value":{"_col0":"8AF163CA6F","_col1":"2011-07-27
>
> 22:48:52","_col2":129390185139228,"_col3":2006,"_col4":4100,"_col5":"10017388=6","_col6":1063,"_col7":"NULL","_col8":"address.com","_col9":"NULL","_col10":"NULL"},"alias":0}
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Cannot
> initialize ScriptOperator
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:320)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
> at
> org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
> ... 7 more
> Caused by: java.io.IOException: Cannot run program "/usr/bin/perl":
> java.io.IOException: error=7, Argument list too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:279)
> ... 15 more
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument list
> too long
> at java.lang.UNIXProcess.(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 16 more
> It seems to me, I found the cause. ScriptOperator.java puts a lot of configs
> as environment variables to the child reduce process. One of variables is
> mapred.input.dir, which in my case more than 150KB. There are a huge amount
> of input directories in this variable. In short, the problem is that Linux
> (up to 2.6.23 kernel version) limits summary size of environment variables
> for child processes to 132KB. This problem could be solved by upgrading the
> kernel. But strings limitations still be 132KB per string in environment
> variable. So such huge variable doesn't work even on my home computer
> (2.6.32). You can read more information on
> (http://www.kernel.org/doc/man-pages/online/pages/man2/execve.2.html).
> For now all our work has been stopped because of this problem and I can't
> find the solution. The only solution, which seems to me more reasonable is to
> get rid of this variable in reducers.
--
This message is automatically generated by JIRA
[jira] [Updated] (HIVE-2759) Change global_limit.q into linux format file
[ https://issues.apache.org/jira/browse/HIVE-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2759: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Zhenxiao > Change global_limit.q into linux format file > > > Key: HIVE-2759 > URL: https://issues.apache.org/jira/browse/HIVE-2759 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Zhenxiao Luo >Assignee: Zhenxiao Luo > Attachments: HIVE-2759.D1497.1.patch, HIVE-2759.D1497.1.patch > > > global_limit.q is in dos format, and has "\ No newline at end of file". > Needs to be changed into linux format, removing all "^M" at the end of each > line. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1248 - Still Failing
Changes for Build #1204 [cws] HIVE-2719. Revert HIVE-2589 (He Yongqiang via cws) Changes for Build #1205 Changes for Build #1207 [namit] HIVE-2718 NPE in union followed by join (He Yongqiang via namit) Changes for Build #1208 Changes for Build #1209 Changes for Build #1210 [namit] HIVE-2674 get_partitions_ps throws TApplicationException if table doesn't exist (Kevin Wilfong via namit) Changes for Build #1211 [cws] HIVE-2203. Extend concat_ws() UDF to support arrays of strings (Zhenxiao Luo via cws) [cws] HIVE-2279. Implement sort(array) UDF (Zhenxiao Luo via cws) Changes for Build #1212 [hashutosh] HIVE-2589 : Newly created partition should inherit properties from table (Ashutosh Chauhan) Changes for Build #1213 Changes for Build #1214 Changes for Build #1215 Changes for Build #1216 [cws] HIVE-2724. Remove unused lib/log4j-1.2.15.jar (Andrew Bayer via cws) Changes for Build #1217 Changes for Build #1218 Changes for Build #1219 [hashutosh] HIVE-2665 : Support for metastore service specific HADOOP_OPTS environment setting (thw via hashutosh) Changes for Build #1220 [namit] HIVE-2727 add a testcase for partitioned view on union and base tables have index (He Yongqiang via namit) Changes for Build #1221 [hashutosh] HIVE-2746 : Metastore client doesn't log properly in case of connection failure to server (hashutosh) [cws] HIVE-2698 [jira] Enable Hadoop-1.0.0 in Hive (Enis Söztutar via Carl Steinbach) Summary: third version of the patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach CC: cwsteinbach, enis Differential Revision: https://reviews.facebook.net/D1389 Changes for Build #1222 [namit] HIVE-2750 Hive multi group by single reducer optimization causes invalid column reference error (Kevin Wilfong via namit) Changes for Build #1223 Changes for Build #1224 [cws] HIVE-2734 [jira] Fix some nondeterministic test output (Zhenxiao Luo via Carl Steinbach) Summary: HIVE-2734: Fix some nondeterministic test output Many Hive query tests lack an ORDER BY clause, and consequently the ordering of the rows in the result set is nondeterministic: groupby1_limit input11_limit input1_limit input_lazyserde join18_multi_distinct join_1to1 join_casesensitive join_filters join_nulls merge3 rcfile_columnar rcfile_lazydecompress rcfile_union sample10 udf_sentences union24 columnarserde_create_shortcut combine1 global_limit Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach CC: zhenxiao, cwsteinbach Differential Revision: https://reviews.facebook.net/D1449 [namit] HIVE-2754 NPE in union with lateral view (Yongqiang He via namit) Changes for Build #1225 Changes for Build #1226 Changes for Build #1227 [namit] HIVE-2755 union follwowed by union_subq does not work if the subquery union has reducers (He Yongqiang via namit) Changes for Build #1228 Changes for Build #1229 [hashutosh] HIVE-2735: PlanUtils.configureTableJobPropertiesForStorageHandler() is not called for partitioned table (sushanth via ashutosh) Changes for Build #1230 [cws] HIVE-2760 [jira] TestCliDriver should log elapsed time Summary: HIVE-2760. TestCliDriver should log elapsed time Test Plan: EMPTY Reviewers: JIRA, ashutoshc Reviewed By: ashutoshc CC: ashutoshc, cwsteinbach Differential Revision: https://reviews.facebook.net/D1503 [cws] HIVE-2662 [jira] Add Ant configuration property for dumping classpath of tests Summary: HIVE-2662. Add Ant configuration property for dumping classpath of tests Test Plan: EMPTY Reviewers: JIRA, jsichi, ashutoshc Reviewed By: ashutoshc CC: ashutoshc Differential Revision: https://reviews.facebook.net/D903 Changes for Build #1231 [hashutosh] HIVE_2645: Hive Web Server startup messages logs incorrect path it is searching for WAR (Edward Capriolo via Ashutosh Chauhan) Changes for Build #1232 Changes for Build #1233 [sdong] HIVE-2249 When creating constant expression for numbers, try to infer type from another comparison operand, instead of trying to use integer first, and then long and double (Zhiqiu Kong via Siying Dong) Changes for Build #1234 Changes for Build #1235 [heyongqiang] HIVE-2765 hbase handler uses ZooKeeperConnectionException which is not compatible with HBase versions other than 0.89 (Pei Yue via He Yongqiang) Changes for Build #1236 Changes for Build #1237 Changes for Build #1238 [heyongqiang] HIVE-2772 [jira] make union31.q deterministic (Namit Jain via Yongqiang He) Summary: https://issues.apache.org/jira/browse/HIVE-2772 HIVE-2772 Test Plan: EMPTY Reviewers: JIRA, ashutoshc Reviewed By: ashutoshc CC: ashutoshc Differential Revision: https://reviews.facebook.net/D1557 [kevinwilfong] HIVE-2758 Metastore is caching too aggressively (Kevin Wilfong reviewed by Carl Steinbach) Changes for Build #1239 Changes for Build #1240 [namit] HIVE-2762 Alter Table Partition Concatenate Fails On Certain
[jira] [Updated] (HIVE-2788) When integrating into MapReduce2, Hive is unable to handle corrupt rcfile archive
[ https://issues.apache.org/jira/browse/HIVE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenxiao Luo updated HIVE-2788: --- Description: archive_corrupt.q is failing due to file format exception when loading archive_corrupt.rc: [junit] Running org.apache.hadoop.hive.cli.TestCliDriver [junit] Begin query: archive_corrupt.q [junit] Copying file: file:/home/cloudera/Code/hive/data/files/archive_corrupt.rc [junit] Exception: Client Execution failed with error code = 9 [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. [junit] junit.framework.AssertionFailedError: Client Execution failed with error code = 9 [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_corrupt(TestCliDriver.java:109) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) [junit] at junit.framework.TestSuite.run(TestSuite.java:238) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs.) [junit] Cleaning up TestCliDriver [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 6.778 sec [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED [for] /home/cloudera/Code/hive/ql/build.xml: The following error occurred while executing this line: [for] /home/cloudera/Code/hive/build.xml:328: The following error occurred while executing this line: [for] /home/cloudera/Code/hive/build-common.xml:453: Tests failed! And, in /build/ql/test/logs/clientpositive/archive_corrupt.q.out: PREHOOK: query: drop table tstsrcpart PREHOOK: type: DROPTABLE POSTHOOK: query: drop table tstsrcpart POSTHOOK: type: DROPTABLE PREHOOK: query: create table tstsrcpart like srcpart PREHOOK: type: CREATETABLE POSTHOOK: query: create table tstsrcpart like srcpart POSTHOOK: type: CREATETABLE POSTHOOK: Output: default@tstsrcpart PREHOOK: query: load data local inpath '../data/files/archive_corrupt.rc' overwrite into table tstsrcpart partition (ds='2008-04-08', hr='11') PREHOOK: type: LOAD PREHOOK: Output: default@tstsrcpart Failed with exception Wrong file format. Please check the file's format. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask When running the following: load data local inpath '../data/files/archive_corrupt.rc' overwrite into table tstsrcpart partition (ds='2008-04-08', hr='11') Get exception: Failed with exception Wrong file format. Please check the file's format. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask was: archive_corrupt.q is failing due to file format exception when loading archive_corrupt.rc: [junit] Running org.apache.hadoop.hive.cli.TestCliDriver [junit] Begin query: archive_corrupt.q [junit] Copying file: file:/home/cloudera/Code/hive/data/files/archive_corrupt.rc [junit] Exception: Client Execution failed with error code = 9 [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. [junit] junit.framework.AssertionFailedError: Client Execution failed with error code = 9 [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_corrupt(TestCliDriver.java:109) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit]
[jira] [Commented] (HIVE-2777) ability to add and drop partitions atomically
[ https://issues.apache.org/jira/browse/HIVE-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203913#comment-13203913 ] Aniket Mokashi commented on HIVE-2777: -- I am adding deleteData option there, deletion of data itself cannot be made atomic. But, its good to have as part of the api. deleteData=true attempts to delete the data. > ability to add and drop partitions atomically > - > > Key: HIVE-2777 > URL: https://issues.apache.org/jira/browse/HIVE-2777 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Aniket Mokashi > Fix For: 0.9.0 > > > Hive should have ability to atomically add and drop partitions. This way > admins can change partitions atomically without breaking the running jobs. It > allows admin to merge several partitions into one. > Essentially, we would like to have an api- add_drop_partitions(String db, > String tbl_name, List addParts, List> dropParts, > boolean deleteData); > This jira covers changes required for metastore and thrift. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2790) When integrating into MapReduce2, GROUP BY on Floating Point and String types are broken
[ https://issues.apache.org/jira/browse/HIVE-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenxiao Luo updated HIVE-2790: --- Description: Query result(-30.33) is missing in the running output: [junit] Begin query: groupby_neg_float.q [junit] 12/01/24 10:57:53 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/24 10:57:53 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/24 10:57:58 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/24 10:57:58 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]*_[0-9]* -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/groupby_neg_float.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/groupby_neg_float.q.out [junit] 14c14,15 [junit] < POSTHOOK: Output: file:/tmp/cloudera/hive_2012-01-24_10-57-51_652_2049459353574360528/-mr-1 [junit] — [junit] > POSTHOOK: Output: file:/data/users/njain/hive5/hive5/build/ql/tmp/815331462/1 [junit] > -30.33 [junit] 28c29,30 [junit] < POSTHOOK: Output: file:/tmp/cloudera/hive_2012-01-24_10-57-56_909_7738265803221872068/-mr-1 [junit] — [junit] > POSTHOOK: Output: file:/data/users/njain/hive5/hive5/build/ql/tmp/2082702070/1 [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_neg_float(TestCliDriver.java:129) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) [junit] at junit.framework.TestSuite.run(TestSuite.java:238) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] Exception: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. [junit] > -30.33 [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs.) When trying the following(GROUP BY on Floating Point type and string type): FROM src SELECT -30.33 GROUP BY -30.33 LIMIT 1; FROM src SELECT '-30.33' GROUP BY '-30.33' LIMIT 1; Get NULL in result was: Query result(-30.33) is missing in the running output: [junit] Begin query: groupby_neg_float.q [junit] 12/01/24 10:57:53 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/24 10:57:53 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/24 10:57:58 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/24 10:57:58 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]*_[0-9]* -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/groupby_neg_float.q.out /home/cloud
[jira] [Commented] (HIVE-2777) ability to add and drop partitions atomically
[ https://issues.apache.org/jira/browse/HIVE-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203896#comment-13203896 ] Ashutosh Chauhan commented on HIVE-2777: So, this api is purely metadata and data won't be touched at all? Data is managed separately in such cases. I think if thats the case, this new api should have an additional parameter bool moveData, which for now works as you described for moveData = false and throws exception for moveData = true. > ability to add and drop partitions atomically > - > > Key: HIVE-2777 > URL: https://issues.apache.org/jira/browse/HIVE-2777 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Aniket Mokashi > Fix For: 0.9.0 > > > Hive should have ability to atomically add and drop partitions. This way > admins can change partitions atomically without breaking the running jobs. It > allows admin to merge several partitions into one. > Essentially, we would like to have an api- add_drop_partitions(String db, > String tbl_name, List addParts, List> dropParts, > boolean deleteData); > This jira covers changes required for metastore and thrift. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[
https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203886#comment-13203886
]
Carl Steinbach commented on HIVE-2782:
--
@Ashutosh: here's a slightly reformatted version of the original query from
ba_table_udfs.q:
{noformat}
SELECT
LENGTH(CAST(src.key AS BINARY)) AS len,
CONCAT(CAST(src.key AS BINARY), CAST(src.value AS BINARY)),
SUBSTR(src.value, 5, 1) AS sub
FROM src
ORDER BY sub
LIMIT 10;
{noformat}
I agree that the orignal query has an order by clause, but this ordering is
being applied to the fifth character of the src.value field, which takes on the
values 1-9. This is not a strong enough condition to guarantee deterministic
ordering of the result set. In the modified version of this query that appears
in the patch I have instead applied the ordering to the src.value field. I also
cast the SUBSTR input as BINARY, which I assume was an oversight in the
original patch.
In this patch I have replaced your original query with this new one:
> New BINARY type produces unexpected results with supported UDFS when using
> MapReduce2
> -
>
> Key: HIVE-2782
> URL: https://issues.apache.org/jira/browse/HIVE-2782
> Project: Hive
> Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Carl Steinbach
> Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch
>
>
> When using MapReduce2 for Hive
> ba_table_udfs is failing with unexpected output:
> [junit] Begin query: ba_table_udfs.q
> [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is
> deprecated. Instead, use mapreduce.jobtracker.system.dir
> [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is
> deprecated. Instead, use mapreduce.cluster.local.dir
> [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I
> lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime
> -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I
> java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I
> Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]*
> more -I job_[0-9]*_[0-9]* -I USING 'java -cp
> /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out
>
> /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out
> [junit] 20,26c20,26
> [junit] < 2 10val_101
> [junit] < 3 164val_164 1
> [junit] < 3 150val_150 1
> [junit] < 2 18val_181
> [junit] < 3 177val_177 1
> [junit] < 2 12val_121
> [junit] < 2 11val_111
> [junit] —
> [junit] > 3 120val_120 1
> [junit] > 3 192val_192 1
> [junit] > 3 119val_119 1
> [junit] > 3 187val_187 1
> [junit] > 3 176val_176 1
> [junit] > 3 199val_199 1
> [junit] > 3 118val_118 1
> [junit] Exception: Client execution results failed with error code = 1
> [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false"
> to get more logs.
> [junit] junit.framework.AssertionFailedError: Client execution results failed
> with error code = 1
> [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false"
> to get more logs.
> [junit] at junit.framework.Assert.fail(Assert.java:50)
> [junit] at
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129)
> [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit] at java.lang.reflect.Method.invoke(Method.java:616)
> [junit] at junit.framework.TestCase.runTest(TestCase.java:168)
> [junit] at junit.framework.TestCase.runBare(TestCase.java:134)
> [junit] at junit.framework.TestResult$1.protect(TestResult.java:110)
> [junit] at junit.framework.TestResult.runProtected(TestResult.java:128)
> [junit] at junit.framework.TestResult.run(TestResult.java:113)
> [junit] at junit.framework.TestCase.run(TestCase.java:124)
> [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243)
> [junit] at junit.framework.TestSuite.run(TestSuite.java:238)
> [junit] at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
> [junit] at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
> [junit] at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
> [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false"
> to get more logs.)
> [junit] Cleaning up TestCliDriver
> [junit] Tests run: 2,
[jira] [Commented] (HIVE-2726) COALESCE(NULL, NULL) NPEs
[ https://issues.apache.org/jira/browse/HIVE-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203850#comment-13203850 ] Namit Jain commented on HIVE-2726: -- The following tests failed: udaf_context_ngrams.q udf_array.q udf_map.q udf_sort_array.q Can you take a look ? For udf_array and udf_map, the output needs to be updated. I haven't looked at the other 2. > COALESCE(NULL, NULL) NPEs > - > > Key: HIVE-2726 > URL: https://issues.apache.org/jira/browse/HIVE-2726 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Chang >Assignee: Jonathan Chang >Priority: Minor > Attachments: HIVE-2726.D1317.1.patch, HIVE-2726.D1317.1.patch > > > ReturnOIResolver returns null when it gets no non-void arguments. This is > bad. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2726) COALESCE(NULL, NULL) NPEs
[ https://issues.apache.org/jira/browse/HIVE-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2726: - Status: Open (was: Patch Available) > COALESCE(NULL, NULL) NPEs > - > > Key: HIVE-2726 > URL: https://issues.apache.org/jira/browse/HIVE-2726 > Project: Hive > Issue Type: Bug >Reporter: Jonathan Chang >Assignee: Jonathan Chang >Priority: Minor > Attachments: HIVE-2726.D1317.1.patch, HIVE-2726.D1317.1.patch > > > ReturnOIResolver returns null when it gets no non-void arguments. This is > bad. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203844#comment-13203844 ] Ashutosh Chauhan commented on HIVE-2782: You modified the original query in non-trivial way. I am not sure if its an ordering issue. Original query had the order-by and its output also was indeed ordered. Further, if you look at the output diff posted by Zhenxiao, its clear that rows are not out of order, but are rather different. This requires further analysis as to why this query is giving wrong results with MR2. > New BINARY type produces unexpected results with supported UDFS when using > MapReduce2 > - > > Key: HIVE-2782 > URL: https://issues.apache.org/jira/browse/HIVE-2782 > Project: Hive > Issue Type: Bug >Reporter: Zhenxiao Luo >Assignee: Carl Steinbach > Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch > > > When using MapReduce2 for Hive > ba_table_udfs is failing with unexpected output: > [junit] Begin query: ba_table_udfs.q > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I > lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime > -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I > java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I > Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* > more -I job_[0-9]*_[0-9]* -I USING 'java -cp > /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out > > /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out > [junit] 20,26c20,26 > [junit] < 2 10val_101 > [junit] < 3 164val_164 1 > [junit] < 3 150val_150 1 > [junit] < 2 18val_181 > [junit] < 3 177val_177 1 > [junit] < 2 12val_121 > [junit] < 2 11val_111 > [junit] — > [junit] > 3 120val_120 1 > [junit] > 3 192val_192 1 > [junit] > 3 119val_119 1 > [junit] > 3 187val_187 1 > [junit] > 3 176val_176 1 > [junit] > 3 199val_199 1 > [junit] > 3 118val_118 1 > [junit] Exception: Client execution results failed with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] junit.framework.AssertionFailedError: Client execution results failed > with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] at junit.framework.Assert.fail(Assert.java:50) > [junit] at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129) > [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [junit] at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > [junit] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [junit] at java.lang.reflect.Method.invoke(Method.java:616) > [junit] at junit.framework.TestCase.runTest(TestCase.java:168) > [junit] at junit.framework.TestCase.runBare(TestCase.java:134) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) > [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) > [junit] at junit.framework.TestResult.run(TestResult.java:113) > [junit] at junit.framework.TestCase.run(TestCase.java:124) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) > [junit] at junit.framework.TestSuite.run(TestSuite.java:238) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs.) > [junit] Cleaning up TestCliDriver > [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.751 sec > [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED > [for] /home/cloudera/Code/hive/ql/build.xml: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build.xml:328: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build-common.xml:453: Tests failed! -- This message is automatically generated by JIRA. If you think it wa
[jira] [Updated] (HIVE-2771) Add support for filter pushdown for key ranges in hbase for keys of type string
[ https://issues.apache.org/jira/browse/HIVE-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2771: -- Attachment: HIVE-2771.D1551.2.patch ashutoshc updated the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string". Reviewers: JIRA, jsichi Added test for key < '20' and key > '30'. REVISION DETAIL https://reviews.facebook.net/D1551 AFFECTED FILES hbase-handler/src/test/results/hbase_ppd_key_range.q.out hbase-handler/src/test/queries/hbase_ppd_key_range.q hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java > Add support for filter pushdown for key ranges in hbase for keys of type > string > --- > > Key: HIVE-2771 > URL: https://issues.apache.org/jira/browse/HIVE-2771 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 0.7.0, 0.7.1, 0.8.0, 0.8.1 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-2771.D1551.1.patch, HIVE-2771.D1551.2.patch > > > This is a subtask of HIVE-1643 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2771) Add support for filter pushdown for key ranges in hbase for keys of type string
[ https://issues.apache.org/jira/browse/HIVE-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203830#comment-13203830 ] Ashutosh Chauhan commented on HIVE-2771: It is. I plan to take that up in a follow-up jira to keep patch sizes small and manageable. Also, I think HIVE-1634 might have impact on that, so I think it make sense to defer that case. > Add support for filter pushdown for key ranges in hbase for keys of type > string > --- > > Key: HIVE-2771 > URL: https://issues.apache.org/jira/browse/HIVE-2771 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 0.7.0, 0.7.1, 0.8.0, 0.8.1 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-2771.D1551.1.patch > > > This is a subtask of HIVE-1643 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2771) Add support for filter pushdown for key ranges in hbase for keys of type string
[ https://issues.apache.org/jira/browse/HIVE-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203812#comment-13203812 ] John Sichi commented on HIVE-2771: -- Isn't a range predicate such as "WHERE key > 20 and key < 30" a fairly important use case? > Add support for filter pushdown for key ranges in hbase for keys of type > string > --- > > Key: HIVE-2771 > URL: https://issues.apache.org/jira/browse/HIVE-2771 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 0.7.0, 0.7.1, 0.8.0, 0.8.1 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-2771.D1551.1.patch > > > This is a subtask of HIVE-1643 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2771) Add support for filter pushdown for key ranges in hbase for keys of type string
[ https://issues.apache.org/jira/browse/HIVE-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203786#comment-13203786 ] Phabricator commented on HIVE-2771: --- ashutoshc has commented on the revision "HIVE-2771 [jira] Add support for filter pushdown for key ranges in hbase for keys of type string". > Also, note the comment in HBaseStorageHandler.decomposePredicate. Have you a tested a case such as WHERE key > 20 and key < 30? For those queries filter will not be pushed exactly because of that comment. I will update the patch with this test where filter is not pushed in HBase and filtering is done by Hive. INLINE COMMENTS hbase-handler/src/test/results/hbase_ppd_key_range.q.out:47 Hive did remove the redundant filter. I verified this by adding private logging. Further, even for queries in hbase_pushdown.q if you look at hbase_pushdown.q.out its the same behavior where for filter key = 90, its been pushed up in storage handler still shows up in explain plan. REVISION DETAIL https://reviews.facebook.net/D1551 > Add support for filter pushdown for key ranges in hbase for keys of type > string > --- > > Key: HIVE-2771 > URL: https://issues.apache.org/jira/browse/HIVE-2771 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 0.7.0, 0.7.1, 0.8.0, 0.8.1 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-2771.D1551.1.patch > > > This is a subtask of HIVE-1643 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2372) java.io.IOException: error=7, Argument list too long
[
https://issues.apache.org/jira/browse/HIVE-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203684#comment-13203684
]
Sergey Tryuber commented on HIVE-2372:
--
That's was my initial idea (you even can see that in my comments). But after
that I've changed my mind because I also "hate to add more and more
parameters". This bug affects may be 0.001% of Hive users and I can't figure
out the case of those 0.001% users when 20KB is not enough. Make this limit
less then 20KB also has no sense, because that's not a bottleneck on modern
environment. So, introducing one more useless option, prepare documentation for
it and compel new Hive users to read/understand it...
> java.io.IOException: error=7, Argument list too long
>
>
> Key: HIVE-2372
> URL: https://issues.apache.org/jira/browse/HIVE-2372
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Sergey Tryuber
>Priority: Critical
> Attachments: HIVE-2372.1.patch.txt
>
>
> I execute a huge query on a table with a lot of 2-level partitions. There is
> a perl reducer in my query. Maps worked ok, but every reducer fails with the
> following exception:
> 2011-08-11 04:58:29,865 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> Executing [/usr/bin/perl, , ]
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> tablename=null
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> partname=null
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> alias=null
> 2011-08-11 04:58:29,935 FATAL ExecReducer:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":129390185139228,"reducesinkkey1":"8AF163CA6F"},"value":{"_col0":"8AF163CA6F","_col1":"2011-07-27
>
> 22:48:52","_col2":129390185139228,"_col3":2006,"_col4":4100,"_col5":"10017388=6","_col6":1063,"_col7":"NULL","_col8":"address.com","_col9":"NULL","_col10":"NULL"},"alias":0}
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Cannot
> initialize ScriptOperator
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:320)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
> at
> org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
> ... 7 more
> Caused by: java.io.IOException: Cannot run program "/usr/bin/perl":
> java.io.IOException: error=7, Argument list too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:279)
> ... 15 more
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument list
> too long
> at java.lang.UNIXProcess.(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 16 more
> It seems to me, I found the cause. ScriptOperator.java puts a lot of configs
> as environment variables to the child reduce process. One of variables is
> mapred.input.dir, which in my case more than 150KB. There are a huge amount
> of input directories in this variable. In short, the problem is that Linux
> (up to 2.6.23 kernel version) limits summary size of environment variables
> for child processes to 132KB. This problem could be solved by upgrading the
> kernel. But strings limitations still be 132KB per string in environment
> variable. So such huge variable doesn't work even on my home computer
> (2.6.32). You can read more
[jira] [Updated] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2782: - Assignee: Carl Steinbach (was: Zhenxiao Luo) Status: Patch Available (was: Open) > New BINARY type produces unexpected results with supported UDFS when using > MapReduce2 > - > > Key: HIVE-2782 > URL: https://issues.apache.org/jira/browse/HIVE-2782 > Project: Hive > Issue Type: Bug >Reporter: Zhenxiao Luo >Assignee: Carl Steinbach > Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch > > > When using MapReduce2 for Hive > ba_table_udfs is failing with unexpected output: > [junit] Begin query: ba_table_udfs.q > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I > lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime > -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I > java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I > Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* > more -I job_[0-9]*_[0-9]* -I USING 'java -cp > /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out > > /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out > [junit] 20,26c20,26 > [junit] < 2 10val_101 > [junit] < 3 164val_164 1 > [junit] < 3 150val_150 1 > [junit] < 2 18val_181 > [junit] < 3 177val_177 1 > [junit] < 2 12val_121 > [junit] < 2 11val_111 > [junit] — > [junit] > 3 120val_120 1 > [junit] > 3 192val_192 1 > [junit] > 3 119val_119 1 > [junit] > 3 187val_187 1 > [junit] > 3 176val_176 1 > [junit] > 3 199val_199 1 > [junit] > 3 118val_118 1 > [junit] Exception: Client execution results failed with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] junit.framework.AssertionFailedError: Client execution results failed > with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] at junit.framework.Assert.fail(Assert.java:50) > [junit] at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129) > [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [junit] at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > [junit] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [junit] at java.lang.reflect.Method.invoke(Method.java:616) > [junit] at junit.framework.TestCase.runTest(TestCase.java:168) > [junit] at junit.framework.TestCase.runBare(TestCase.java:134) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) > [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) > [junit] at junit.framework.TestResult.run(TestResult.java:113) > [junit] at junit.framework.TestCase.run(TestCase.java:124) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) > [junit] at junit.framework.TestSuite.run(TestSuite.java:238) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs.) > [junit] Cleaning up TestCliDriver > [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.751 sec > [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED > [for] /home/cloudera/Code/hive/ql/build.xml: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build.xml:328: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build-common.xml:453: Tests failed! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2782: - Attachment: HIVE-2782.D1653.1.patch > New BINARY type produces unexpected results with supported UDFS when using > MapReduce2 > - > > Key: HIVE-2782 > URL: https://issues.apache.org/jira/browse/HIVE-2782 > Project: Hive > Issue Type: Bug >Reporter: Zhenxiao Luo >Assignee: Zhenxiao Luo > Attachments: HIVE-2782.D1653.1.patch, HIVE-2782.D1653.1.patch > > > When using MapReduce2 for Hive > ba_table_udfs is failing with unexpected output: > [junit] Begin query: ba_table_udfs.q > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I > lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime > -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I > java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I > Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* > more -I job_[0-9]*_[0-9]* -I USING 'java -cp > /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out > > /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out > [junit] 20,26c20,26 > [junit] < 2 10val_101 > [junit] < 3 164val_164 1 > [junit] < 3 150val_150 1 > [junit] < 2 18val_181 > [junit] < 3 177val_177 1 > [junit] < 2 12val_121 > [junit] < 2 11val_111 > [junit] — > [junit] > 3 120val_120 1 > [junit] > 3 192val_192 1 > [junit] > 3 119val_119 1 > [junit] > 3 187val_187 1 > [junit] > 3 176val_176 1 > [junit] > 3 199val_199 1 > [junit] > 3 118val_118 1 > [junit] Exception: Client execution results failed with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] junit.framework.AssertionFailedError: Client execution results failed > with error code = 1 > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs. > [junit] at junit.framework.Assert.fail(Assert.java:50) > [junit] at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129) > [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [junit] at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > [junit] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [junit] at java.lang.reflect.Method.invoke(Method.java:616) > [junit] at junit.framework.TestCase.runTest(TestCase.java:168) > [junit] at junit.framework.TestCase.runBare(TestCase.java:134) > [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) > [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) > [junit] at junit.framework.TestResult.run(TestResult.java:113) > [junit] at junit.framework.TestCase.run(TestCase.java:124) > [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) > [junit] at junit.framework.TestSuite.run(TestSuite.java:238) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) > [junit] at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) > [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" > to get more logs.) > [junit] Cleaning up TestCliDriver > [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.751 sec > [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED > [for] /home/cloudera/Code/hive/ql/build.xml: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build.xml:328: The following error occurred > while executing this line: > [for] /home/cloudera/Code/hive/build-common.xml:453: Tests failed! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2782) New BINARY type produces unexpected results with supported UDFS when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2782: -- Attachment: HIVE-2782.D1653.1.patch cwsteinbach requested code review of "HIVE-2782 [jira] New BINARY type produces unexpected results with supported UDFS when using MapReduce2". Reviewers: JIRA HIVE-2782. Make ba_table_udfs.q deterministic When using MapReduce2 for Hive ba_table_udfs is failing with unexpected output: [junit] Begin query: ba_table_udfs.q [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 13:32:28 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]*_[0-9]* -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/ba_table_udfs.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/ba_table_udfs.q.out [junit] 20,26c20,26 [junit] < 2 10val_101 [junit] < 3 164val_164 1 [junit] < 3 150val_150 1 [junit] < 2 18val_181 [junit] < 3 177val_177 1 [junit] < 2 12val_121 [junit] < 2 11val_111 [junit] — [junit] > 3 120val_120 1 [junit] > 3 192val_192 1 [junit] > 3 119val_119 1 [junit] > 3 187val_187 1 [junit] > 3 176val_176 1 [junit] > 3 199val_199 1 [junit] > 3 118val_118 1 [junit] Exception: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs(TestCliDriver.java:129) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) [junit] at junit.framework.TestSuite.run(TestSuite.java:238) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs.) [junit] Cleaning up TestCliDriver [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 10.751 sec [junit] Test org.apache.hadoop.hive.cli.TestCliDriver FAILED [for] /home/cloudera/Code/hive/ql/build.xml: The following error occurred while executing this line: [for] /home/cloudera/Code/hive/build.xml:328: The following error occurred while executing this line: [for] /home/cloudera/Code/hive/build-common.xml:453: Tests failed! TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1653 AFFECTED FILES ql/src/test/queries/clientpositive/ba_table_udfs.q ql/src/test/results/clientpositive/ba_table_udfs.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3531/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > New BINARY type produces unexpected results with supported UDFS when using > MapReduce2 > - > > Key: HIVE-2782 > URL: https://issues.apache.org/jira/browse/HIVE-2782 > Project: Hive > Issue Type: Bug >
[jira] [Updated] (HIVE-2789) When integrating into MapReduce2, 'select * from src distribute by src.key limit 1' returns non-deterministic result
[ https://issues.apache.org/jira/browse/HIVE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2789: - Status: Patch Available (was: Open) > When integrating into MapReduce2, 'select * from src distribute by src.key > limit 1' returns non-deterministic result > > > Key: HIVE-2789 > URL: https://issues.apache.org/jira/browse/HIVE-2789 > Project: Hive > Issue Type: Bug >Reporter: Zhenxiao Luo >Assignee: Carl Steinbach > Attachments: HIVE-2789.D1647.1.patch, HIVE-2789.D1647.1.patch > > > query_properties.q test failure: > [junit] Begin query: query_properties.q > [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 17:00:22 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 17:00:22 WARN conf.Configurati
[jira] [Updated] (HIVE-2789) When integrating into MapReduce2, 'select * from src distribute by src.key limit 1' returns non-deterministic result
[ https://issues.apache.org/jira/browse/HIVE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2789: - Attachment: HIVE-2789.D1647.1.patch > When integrating into MapReduce2, 'select * from src distribute by src.key > limit 1' returns non-deterministic result > > > Key: HIVE-2789 > URL: https://issues.apache.org/jira/browse/HIVE-2789 > Project: Hive > Issue Type: Bug >Reporter: Zhenxiao Luo >Assignee: Carl Steinbach > Attachments: HIVE-2789.D1647.1.patch, HIVE-2789.D1647.1.patch > > > query_properties.q test failure: > [junit] Begin query: query_properties.q > [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.local.dir is > deprecated. Instead, use mapreduce.cluster.local.dir > [junit] 12/01/23 17:00:22 WARN conf.Configuration: mapred.system.dir is > deprecated. Instead, use mapreduce.jobtracker.system.dir > [junit] 12/01/23 17:00:22 WARN conf.Configuratio
[jira] [Updated] (HIVE-2789) When integrating into MapReduce2, 'select * from src distribute by src.key limit 1' returns non-deterministic result
[ https://issues.apache.org/jira/browse/HIVE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2789: -- Attachment: HIVE-2789.D1647.1.patch cwsteinbach requested code review of "HIVE-2789 [jira] When integrating into MapReduce2, 'select * from src distribute by src.key limit 1' returns non-deterministic result". Reviewers: JIRA Fix nondeterminism in query_properties.q by limiting all result sets to 0 rows. query_properties.q test failure: [junit] Begin query: query_properties.q [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:22 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:22 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:27 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:27 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit]
[jira] [Commented] (HIVE-494) Select columns by index instead of name
[ https://issues.apache.org/jira/browse/HIVE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203357#comment-13203357 ] Phabricator commented on HIVE-494: -- navis has commented on the revision "HIVE-494 [jira] Select columns by index instead of name". Simple patch, but was very useful for me to implement query generator for random-forest. REVISION DETAIL https://reviews.facebook.net/D1641 > Select columns by index instead of name > --- > > Key: HIVE-494 > URL: https://issues.apache.org/jira/browse/HIVE-494 > Project: Hive > Issue Type: Wish > Components: Clients, Query Processor >Reporter: Adam Kramer >Priority: Minor > Labels: SQL > Attachments: HIVE-494.D1641.1.patch > > > SELECT mytable[0], mytable[2] FROM some_table_name mytable; > ...should return the first and third columns, respectively, from mytable > regardless of their column names. > The need for "names" specifically is kind of silly when they just get > translated into numbers anyway. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-494) Select columns by index instead of name
[ https://issues.apache.org/jira/browse/HIVE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-494: - Attachment: HIVE-494.D1641.1.patch navis requested code review of "HIVE-494 [jira] Select columns by index instead of name". Reviewers: JIRA DPAL-732 Select columns by index instead of name SELECT mytable[0], mytable[2] FROM some_table_name mytable; ...should return the first and third columns, respectively, from mytable regardless of their column names. The need for "names" specifically is kind of silly when they just get translated into numbers anyway. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1641 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java ql/src/test/queries/clientpositive/select_by_column_index.q ql/src/test/results/clientpositive/select_by_column_index.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3501/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Select columns by index instead of name > --- > > Key: HIVE-494 > URL: https://issues.apache.org/jira/browse/HIVE-494 > Project: Hive > Issue Type: Wish > Components: Clients, Query Processor >Reporter: Adam Kramer >Priority: Minor > Labels: SQL > Attachments: HIVE-494.D1641.1.patch > > > SELECT mytable[0], mytable[2] FROM some_table_name mytable; > ...should return the first and third columns, respectively, from mytable > regardless of their column names. > The need for "names" specifically is kind of silly when they just get > translated into numbers anyway. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
