[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[ https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742857#comment-13742857 ] Hudson commented on HIVE-5105: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #62 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/62/]) HIVE-5105 HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap (Eugene Koifman via Sushanth Sowmyan) (khorgath: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514929) * /hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/data/schema/HCatSchema.java * /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/data/schema/TestHCatSchema.java HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap - Key: HIVE-5105 URL: https://issues.apache.org/jira/browse/HIVE-5105 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.12.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 0.12.0 Attachments: HIVE-5105.patch org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema hcatFieldSchema) makes the following call: fieldPositionMap.remove(hcatFieldSchema); but fieldPositionMap is of type MapString, Integer so the element is not getting removed Here's a detailed comment from [~sushanth] The result is that that the name will not be removed from fieldPositionMap. This results in 2 things: a) If anyone tries to append a field to a hcatschema after having removed that field, it shouldn't fail, but it will. b) If anyone asks for the position of the removed field by name, it will still give the position. Now, there is only one place in hcat code where we remove a field, and that is called from HCatOutputFormat.setSchema, where we try to detect if the user specified partition column names in the schema when they shouldn't have, and if they did, we remove it. Normally, people do not specify this, and this check tends to be superfluous. Once we do this, we wind up serializing that new object (after performing some validations), and this does appear to stay through the serialization (and eventual deserialization) which is very worrying. However, we are luckily saved by the fact that we do not append that field to it at any time(all appends in hcat code are done on newly initialized HCatSchema objects which have had no removes done on them), and we don't ask for the position of something we do not expect to be there(harder to verify for certain, but seems to be the case on inspection). The main part that gives me worry is that HCatSchema is part of our public interface for HCat, in that M/R programs that use HCat can use it, and thus, they might have more interesting usage patterns that are hitting this bug. I can't think of any currently open bugs that is caused by this because of the rarity of the situation, but nevertheless, something we should fix immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4838) Refactor MapJoin HashMap code to improve testability and readability
[ https://issues.apache.org/jira/browse/HIVE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742863#comment-13742863 ] Hudson commented on HIVE-4838: -- ABORTED: Integrated in Hive-trunk-hadoop2 #365 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/365/]) HIVE-4838 : Refactor MapJoin HashMap code to improve testability and readability (Brock Noland via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514760) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinMetaData.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinKey.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractRowContainer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/DCLLItem.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MRU.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinDoubleKeys.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectKey.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectSerDeContext.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinObjectValue.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinSingleKey.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestHashMapWrapper.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/mapjoin * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/mapjoin/TestMapJoinMemoryExhaustionHandler.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinKey.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinKeys.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinRowContainer.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/Utilities.java Refactor MapJoin HashMap code to improve testability and readability Key: HIVE-4838 URL: https://issues.apache.org/jira/browse/HIVE-4838 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.12.0 Attachments: HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch MapJoin is an essential component for high performance joins in Hive and the current code has done great service for many years. However, the code is showing it's age and currently suffers from the following issues: * Uses static state via the MapJoinMetaData class to pass serialization metadata to the Key, Row classes. * The api of a logical Table Container is not defined and therefore it's unclear what apis HashMapWrapper needs to publicize. Additionally HashMapWrapper has many used public methods. * HashMapWrapper contains logic to serialize, test memory bounds, and implement the table container. Ideally these logical units could be
[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[ https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742877#comment-13742877 ] Hudson commented on HIVE-5105: -- FAILURE: Integrated in Hive-trunk-hadoop1-ptest #131 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/131/]) HIVE-5105 HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap (Eugene Koifman via Sushanth Sowmyan) (khorgath: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514929) * /hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/data/schema/HCatSchema.java * /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/data/schema/TestHCatSchema.java HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap - Key: HIVE-5105 URL: https://issues.apache.org/jira/browse/HIVE-5105 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.12.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 0.12.0 Attachments: HIVE-5105.patch org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema hcatFieldSchema) makes the following call: fieldPositionMap.remove(hcatFieldSchema); but fieldPositionMap is of type MapString, Integer so the element is not getting removed Here's a detailed comment from [~sushanth] The result is that that the name will not be removed from fieldPositionMap. This results in 2 things: a) If anyone tries to append a field to a hcatschema after having removed that field, it shouldn't fail, but it will. b) If anyone asks for the position of the removed field by name, it will still give the position. Now, there is only one place in hcat code where we remove a field, and that is called from HCatOutputFormat.setSchema, where we try to detect if the user specified partition column names in the schema when they shouldn't have, and if they did, we remove it. Normally, people do not specify this, and this check tends to be superfluous. Once we do this, we wind up serializing that new object (after performing some validations), and this does appear to stay through the serialization (and eventual deserialization) which is very worrying. However, we are luckily saved by the fact that we do not append that field to it at any time(all appends in hcat code are done on newly initialized HCatSchema objects which have had no removes done on them), and we don't ask for the position of something we do not expect to be there(harder to verify for certain, but seems to be the case on inspection). The main part that gives me worry is that HCatSchema is part of our public interface for HCat, in that M/R programs that use HCat can use it, and thus, they might have more interesting usage patterns that are hitting this bug. I can't think of any currently open bugs that is caused by this because of the rarity of the situation, but nevertheless, something we should fix immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[ https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742894#comment-13742894 ] Hudson commented on HIVE-5105: -- FAILURE: Integrated in Hive-trunk-h0.21 #2274 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2274/]) HIVE-5105 HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap (Eugene Koifman via Sushanth Sowmyan) (khorgath: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514929) * /hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/data/schema/HCatSchema.java * /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/data/schema/TestHCatSchema.java HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap - Key: HIVE-5105 URL: https://issues.apache.org/jira/browse/HIVE-5105 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.12.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 0.12.0 Attachments: HIVE-5105.patch org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema hcatFieldSchema) makes the following call: fieldPositionMap.remove(hcatFieldSchema); but fieldPositionMap is of type MapString, Integer so the element is not getting removed Here's a detailed comment from [~sushanth] The result is that that the name will not be removed from fieldPositionMap. This results in 2 things: a) If anyone tries to append a field to a hcatschema after having removed that field, it shouldn't fail, but it will. b) If anyone asks for the position of the removed field by name, it will still give the position. Now, there is only one place in hcat code where we remove a field, and that is called from HCatOutputFormat.setSchema, where we try to detect if the user specified partition column names in the schema when they shouldn't have, and if they did, we remove it. Normally, people do not specify this, and this check tends to be superfluous. Once we do this, we wind up serializing that new object (after performing some validations), and this does appear to stay through the serialization (and eventual deserialization) which is very worrying. However, we are luckily saved by the fact that we do not append that field to it at any time(all appends in hcat code are done on newly initialized HCatSchema objects which have had no removes done on them), and we don't ask for the position of something we do not expect to be there(harder to verify for certain, but seems to be the case on inspection). The main part that gives me worry is that HCatSchema is part of our public interface for HCat, in that M/R programs that use HCat can use it, and thus, they might have more interesting usage patterns that are hitting this bug. I can't think of any currently open bugs that is caused by this because of the rarity of the situation, but nevertheless, something we should fix immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4388) HBase tests fail against Hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742925#comment-13742925 ] Brock Noland commented on HIVE-4388: [~stack] looks like the hbase.defaults.for.version in hbase-common-0.95.2-hadoop1.jar is 0.95.2-hadoop2. HBase tests fail against Hadoop 2 - Key: HIVE-4388 URL: https://issues.apache.org/jira/browse/HIVE-4388 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Gunther Hagleitner Assignee: Brock Noland Attachments: HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388-wip.txt Currently we're building by default against 0.92. When you run against hadoop 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963. HIVE-3861 upgrades the version of hbase used. This will get you past the problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-5100) RCFile::sync(long) missing 1 byte in System.arraycopy()
[ https://issues.apache.org/jira/browse/HIVE-5100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-5100: - Assignee: Gopal V RCFile::sync(long) missing 1 byte in System.arraycopy() - Key: HIVE-5100 URL: https://issues.apache.org/jira/browse/HIVE-5100 Project: Hive Issue Type: Bug Reporter: tagus wang Assignee: Gopal V this has a bug in this: System.arraycopy(buffer, buffer.length - prefix - 1, buffer, 0, prefix); it should be System.arraycopy(buffer, buffer.length - prefix, buffer, 0, prefix); it is missing 1 byte at the end. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4940) udaf_percentile_approx.q is not deterministic
[ https://issues.apache.org/jira/browse/HIVE-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13743032#comment-13743032 ] Hudson commented on HIVE-4940: -- ABORTED: Integrated in Hive-trunk-hadoop2 #366 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/366/]) HIVE-4940 udaf_percentile_approx.q is not deterministic (Navis via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514771) * /hive/trunk/ql/src/test/queries/clientpositive/udaf_percentile_approx.q * /hive/trunk/ql/src/test/queries/clientpositive/udaf_percentile_approx_20.q * /hive/trunk/ql/src/test/queries/clientpositive/udaf_percentile_approx_23.q * /hive/trunk/ql/src/test/results/clientpositive/udaf_percentile_approx.q.out * /hive/trunk/ql/src/test/results/clientpositive/udaf_percentile_approx_20.q.out * /hive/trunk/ql/src/test/results/clientpositive/udaf_percentile_approx_23.q.out udaf_percentile_approx.q is not deterministic - Key: HIVE-4940 URL: https://issues.apache.org/jira/browse/HIVE-4940 Project: Hive Issue Type: Sub-task Components: Tests Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4940.D12189.1.patch Makes different result for 20(S) and 23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[ https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13743031#comment-13743031 ] Hudson commented on HIVE-5105: -- ABORTED: Integrated in Hive-trunk-hadoop2 #366 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/366/]) HIVE-5105 HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap (Eugene Koifman via Sushanth Sowmyan) (khorgath: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514929) * /hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/data/schema/HCatSchema.java * /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/data/schema/TestHCatSchema.java HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap - Key: HIVE-5105 URL: https://issues.apache.org/jira/browse/HIVE-5105 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.12.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 0.12.0 Attachments: HIVE-5105.patch org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema hcatFieldSchema) makes the following call: fieldPositionMap.remove(hcatFieldSchema); but fieldPositionMap is of type MapString, Integer so the element is not getting removed Here's a detailed comment from [~sushanth] The result is that that the name will not be removed from fieldPositionMap. This results in 2 things: a) If anyone tries to append a field to a hcatschema after having removed that field, it shouldn't fail, but it will. b) If anyone asks for the position of the removed field by name, it will still give the position. Now, there is only one place in hcat code where we remove a field, and that is called from HCatOutputFormat.setSchema, where we try to detect if the user specified partition column names in the schema when they shouldn't have, and if they did, we remove it. Normally, people do not specify this, and this check tends to be superfluous. Once we do this, we wind up serializing that new object (after performing some validations), and this does appear to stay through the serialization (and eventual deserialization) which is very worrying. However, we are luckily saved by the fact that we do not append that field to it at any time(all appends in hcat code are done on newly initialized HCatSchema objects which have had no removes done on them), and we don't ask for the position of something we do not expect to be there(harder to verify for certain, but seems to be the case on inspection). The main part that gives me worry is that HCatSchema is part of our public interface for HCat, in that M/R programs that use HCat can use it, and thus, they might have more interesting usage patterns that are hitting this bug. I can't think of any currently open bugs that is caused by this because of the rarity of the situation, but nevertheless, something we should fix immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5115) Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2
Deepesh Khandelwal created HIVE-5115: Summary: Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2 Key: HIVE-5115 URL: https://issues.apache.org/jira/browse/HIVE-5115 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.12.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal In the webhcat e2e testsuite we have two MR job submission tests TestMapReduce_1 (in jobsubmission.conf) runs the hadoop wordcount example. Intention of this one is to test MR job submission using WebHCat. TestHeartbeat_2 (in jobsubmission2.conf) runs the hadoop sleep example. Intention of this one is to test a long running (10min) WebHCat MR job, see HIVE-4808. In Hadoop 1, both of these example MR applications are packaged in hadoop-examples.jar In Hadoop 2, sleep job is bundled in hadoop-mapreduce-client-jobclient.jar and wordcount is bundled in hadoop-mapreduce-examples.jar Currently the webhcat tests assume that both these MR applications are in one jar that we copy as hexamples.jar. To run these against Hadoop 2 I can think of three simple solutions: (1) Stick with one jar and run sleep application in the TestMapReduce_1 test as well. (2) Eliminate the test TestMapReduce_1 as TestHeartbeat_2 runs a MR job as well. (3) Require two different jars for Hadoop 2 and call them hclient.jar (containing sleep application) and hexamples.jar (containing wordcount application). For Hadoop 1, we would make two copies of the same hadoop-examples.jar application and call them hsleep.jar and examples.jar. The three approaches mentioned here would require least of changes. My inclination is towards (2). Let me know what you think and I can provide the patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5115) Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-5115: - Component/s: (was: WebHCat) Tests Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2 -- Key: HIVE-5115 URL: https://issues.apache.org/jira/browse/HIVE-5115 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.12.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal In the webhcat e2e testsuite we have two MR job submission tests TestMapReduce_1 (in jobsubmission.conf) runs the hadoop wordcount example. Intention of this one is to test MR job submission using WebHCat. TestHeartbeat_2 (in jobsubmission2.conf) runs the hadoop sleep example. Intention of this one is to test a long running (10min) WebHCat MR job, see HIVE-4808. In Hadoop 1, both of these example MR applications are packaged in hadoop-examples.jar In Hadoop 2, sleep job is bundled in hadoop-mapreduce-client-jobclient.jar and wordcount is bundled in hadoop-mapreduce-examples.jar Currently the webhcat tests assume that both these MR applications are in one jar that we copy as hexamples.jar. To run these against Hadoop 2 I can think of three simple solutions: (1) Stick with one jar and run sleep application in the TestMapReduce_1 test as well. (2) Eliminate the test TestMapReduce_1 as TestHeartbeat_2 runs a MR job as well. (3) Require two different jars for Hadoop 2 and call them hclient.jar (containing sleep application) and hexamples.jar (containing wordcount application). For Hadoop 1, we would make two copies of the same hadoop-examples.jar application and call them hsleep.jar and examples.jar. The three approaches mentioned here would require least of changes. My inclination is towards (2). Let me know what you think and I can provide the patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4388) HBase tests fail against Hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13743075#comment-13743075 ] stack commented on HIVE-4388: - [~brocknoland] Hmm Just ran through the build and seems to put the right value in there. I must have mangled the build somehow. Want me to regen? HBase tests fail against Hadoop 2 - Key: HIVE-4388 URL: https://issues.apache.org/jira/browse/HIVE-4388 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Gunther Hagleitner Assignee: Brock Noland Attachments: HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388-wip.txt Currently we're building by default against 0.92. When you run against hadoop 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963. HIVE-3861 upgrades the version of hbase used. This will get you past the problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5115) Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13743083#comment-13743083 ] Thejas M Nair commented on HIVE-5115: - I think it is useful to have a MR test that reads an input file and writes an ouput. It can help when we are validating things like the proxy-user functionality. So I would go for option 3. Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2 -- Key: HIVE-5115 URL: https://issues.apache.org/jira/browse/HIVE-5115 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.12.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal In the webhcat e2e testsuite we have two MR job submission tests TestMapReduce_1 (in jobsubmission.conf) runs the hadoop wordcount example. Intention of this one is to test MR job submission using WebHCat. TestHeartbeat_2 (in jobsubmission2.conf) runs the hadoop sleep example. Intention of this one is to test a long running (10min) WebHCat MR job, see HIVE-4808. In Hadoop 1, both of these example MR applications are packaged in hadoop-examples.jar In Hadoop 2, sleep job is bundled in hadoop-mapreduce-client-jobclient.jar and wordcount is bundled in hadoop-mapreduce-examples.jar Currently the webhcat tests assume that both these MR applications are in one jar that we copy as hexamples.jar. To run these against Hadoop 2 I can think of three simple solutions: (1) Stick with one jar and run sleep application in the TestMapReduce_1 test as well. (2) Eliminate the test TestMapReduce_1 as TestHeartbeat_2 runs a MR job as well. (3) Require two different jars for Hadoop 2 and call them hclient.jar (containing sleep application) and hexamples.jar (containing wordcount application). For Hadoop 1, we would make two copies of the same hadoop-examples.jar application and call them hsleep.jar and examples.jar. The three approaches mentioned here would require least of changes. My inclination is towards (2). Let me know what you think and I can provide the patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4388) HBase tests fail against Hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13743103#comment-13743103 ] Brock Noland commented on HIVE-4388: That'd be great as out test fail with: hbase-default.xml file seems to be for and old version of HBase (0.95.2-hadoop2), this version is 0.95.2-hadoop1 HBase tests fail against Hadoop 2 - Key: HIVE-4388 URL: https://issues.apache.org/jira/browse/HIVE-4388 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Gunther Hagleitner Assignee: Brock Noland Attachments: HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388-wip.txt Currently we're building by default against 0.92. When you run against hadoop 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963. HIVE-3861 upgrades the version of hbase used. This will get you past the problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5115) Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13743124#comment-13743124 ] Eugene Koifman commented on HIVE-5115: -- I also think 3 is better long term as we add more/different tests in the future. Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2 -- Key: HIVE-5115 URL: https://issues.apache.org/jira/browse/HIVE-5115 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.12.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal In the webhcat e2e testsuite we have two MR job submission tests TestMapReduce_1 (in jobsubmission.conf) runs the hadoop wordcount example. Intention of this one is to test MR job submission using WebHCat. TestHeartbeat_2 (in jobsubmission2.conf) runs the hadoop sleep example. Intention of this one is to test a long running (10min) WebHCat MR job, see HIVE-4808. In Hadoop 1, both of these example MR applications are packaged in hadoop-examples.jar In Hadoop 2, sleep job is bundled in hadoop-mapreduce-client-jobclient.jar and wordcount is bundled in hadoop-mapreduce-examples.jar Currently the webhcat tests assume that both these MR applications are in one jar that we copy as hexamples.jar. To run these against Hadoop 2 I can think of three simple solutions: (1) Stick with one jar and run sleep application in the TestMapReduce_1 test as well. (2) Eliminate the test TestMapReduce_1 as TestHeartbeat_2 runs a MR job as well. (3) Require two different jars for Hadoop 2 and call them hclient.jar (containing sleep application) and hexamples.jar (containing wordcount application). For Hadoop 1, we would make two copies of the same hadoop-examples.jar application and call them hsleep.jar and examples.jar. The three approaches mentioned here would require least of changes. My inclination is towards (2). Let me know what you think and I can provide the patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5115) Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-5115: - Status: Patch Available (was: Open) Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2 -- Key: HIVE-5115 URL: https://issues.apache.org/jira/browse/HIVE-5115 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.12.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Attachments: HIVE-5115.patch In the webhcat e2e testsuite we have two MR job submission tests TestMapReduce_1 (in jobsubmission.conf) runs the hadoop wordcount example. Intention of this one is to test MR job submission using WebHCat. TestHeartbeat_2 (in jobsubmission2.conf) runs the hadoop sleep example. Intention of this one is to test a long running (10min) WebHCat MR job, see HIVE-4808. In Hadoop 1, both of these example MR applications are packaged in hadoop-examples.jar In Hadoop 2, sleep job is bundled in hadoop-mapreduce-client-jobclient.jar and wordcount is bundled in hadoop-mapreduce-examples.jar Currently the webhcat tests assume that both these MR applications are in one jar that we copy as hexamples.jar. To run these against Hadoop 2 I can think of three simple solutions: (1) Stick with one jar and run sleep application in the TestMapReduce_1 test as well. (2) Eliminate the test TestMapReduce_1 as TestHeartbeat_2 runs a MR job as well. (3) Require two different jars for Hadoop 2 and call them hclient.jar (containing sleep application) and hexamples.jar (containing wordcount application). For Hadoop 1, we would make two copies of the same hadoop-examples.jar application and call them hsleep.jar and examples.jar. The three approaches mentioned here would require least of changes. My inclination is towards (2). Let me know what you think and I can provide the patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5115) Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2
[ https://issues.apache.org/jira/browse/HIVE-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-5115: - Attachment: HIVE-5115.patch Thanks [~thejas]. Attached the patch with the suggested change. Webhcat e2e tests TestMapReduce_1 and TestHeartbeat_2 require changes for Hadoop 2 -- Key: HIVE-5115 URL: https://issues.apache.org/jira/browse/HIVE-5115 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.12.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Attachments: HIVE-5115.patch In the webhcat e2e testsuite we have two MR job submission tests TestMapReduce_1 (in jobsubmission.conf) runs the hadoop wordcount example. Intention of this one is to test MR job submission using WebHCat. TestHeartbeat_2 (in jobsubmission2.conf) runs the hadoop sleep example. Intention of this one is to test a long running (10min) WebHCat MR job, see HIVE-4808. In Hadoop 1, both of these example MR applications are packaged in hadoop-examples.jar In Hadoop 2, sleep job is bundled in hadoop-mapreduce-client-jobclient.jar and wordcount is bundled in hadoop-mapreduce-examples.jar Currently the webhcat tests assume that both these MR applications are in one jar that we copy as hexamples.jar. To run these against Hadoop 2 I can think of three simple solutions: (1) Stick with one jar and run sleep application in the TestMapReduce_1 test as well. (2) Eliminate the test TestMapReduce_1 as TestHeartbeat_2 runs a MR job as well. (3) Require two different jars for Hadoop 2 and call them hclient.jar (containing sleep application) and hexamples.jar (containing wordcount application). For Hadoop 1, we would make two copies of the same hadoop-examples.jar application and call them hsleep.jar and examples.jar. The three approaches mentioned here would require least of changes. My inclination is towards (2). Let me know what you think and I can provide the patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
