Re: Write access to Hive twiki
Hi Thiruvel, You should now have edit permissions. Please email me if you run into any problems. Thanks. Carl On Thu, Mar 8, 2012 at 1:14 AM, Thiruvel Thirumoolan thiru...@yahoo-inc.com wrote: Ping. My user-id is 'thiruvel' and I am requesting edit permissions on 'https://cwiki.apache.org/confluence/display/Hive' Thanks, Thiruvel On 3/7/12 2:29 PM, Thiruvel Thirumoolan thiru...@yahoo-inc.com wrote: Hello, Can you provide me write access to Hive twiki? As much as possible, we would like to rely on documentation on Apache and not maintain a twiki internally. Thanks, Thiruvel
[jira] [Created] (HIVE-2855) Archived tables/partitions use malformed HAR URIs when run in local mode
Archived tables/partitions use malformed HAR URIs when run in local mode Key: HIVE-2855 URL: https://issues.apache.org/jira/browse/HIVE-2855 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2855) Archived tables/partitions use malformed HAR URIs when run in local mode
[ https://issues.apache.org/jira/browse/HIVE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2855: -- Attachment: HIVE-2855.D2187.1.patch cwsteinbach requested code review of HIVE-2855 [jira] Archived tables/partitions use malformed HAR URIs when run in local mode. Reviewers: JIRA HIVE-2855. Archived tables/partitions use malformed HAR URIs when run in local mode TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2187 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4791/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Archived tables/partitions use malformed HAR URIs when run in local mode Key: HIVE-2855 URL: https://issues.apache.org/jira/browse/HIVE-2855 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-2855.D2187.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2837) insert into external tables should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225286#comment-13225286 ] Alan Gates commented on HIVE-2837: -- If this is to be added it should definitely be configurable, with the default off, since this is a major backwards incompatibility. insert into external tables should not be allowed - Key: HIVE-2837 URL: https://issues.apache.org/jira/browse/HIVE-2837 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain This is a very risky thing to allow. Since, the external tables can point to any user location, which can potentially corrupt some other tables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
HIVE and S3
Hi, I had to post this question to this list because I feel there might be a bug here. I'm having problems with HIVE- EC2 reading files on S3 written by other tools I have a lot of files and folders on S3 created by s3cmd and utilized by Elastic MapReduce (HIVE) and they work interchangeably, files created by HIVE-EMR can be read by s3cmd and vice versa. However, I'm having problems with HIVE/Hadoop running on EC2. Both Hive 0.7 and 0.8 seem to create an additional folder / on S3 For example, if I have a file s3://bucket/path/0 created by s3cmd or HIVE-EMR and I try to create an external table on HIVE- EC2 create external table wc(site string, cnt int) row format delimited fields terminated by '\t' stored as textfile location 's3://bucket/path' This does not recognize the EMR created s3 folders, instead I see a new folder / bucket / / / path When I look at the debug information, HIVE seems to be sending an extra / when creating a table Here is a debug message and if you see the path, there is a / and a %2f. Probably a bug in the code ? hive create external table wc(site string, cnt int) location 's3://masked/wcoverlay/'; StringToSignGETWed, 07 Mar 2012 18:26:03 GMT/masked/%2Fwcoverlay/StringToSignAWSAccessKeyId. Am I missing something? Thanks, Balaji
[jira] [Updated] (HIVE-2838) cleanup readentity/writeentity
[ https://issues.apache.org/jira/browse/HIVE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2838: -- Attachment: HIVE-2838.D2193.1.patch njain requested code review of HIVE-2838 [jira] cleanup readentity/writeentity. Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2838 HIVE-2838 Cleaup read/write Entities Ideally, there should be one common entity instead of readentity/writeentity. Unfortunately, that would be a backward incompatible change since users os hive might have written there own hooks, where they are using readentity/writeentity. We should atleast create a common class, and then we can deprecate read/write entity later, for a new release. For now, I propose to make a backward compatible change. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2193 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4815/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. cleanup readentity/writeentity -- Key: HIVE-2838 URL: https://issues.apache.org/jira/browse/HIVE-2838 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: HIVE-2838.D2193.1.patch Ideally, there should be one common entity instead of readentity/writeentity. Unfortunately, that would be a backward incompatible change since users os hive might have written there own hooks, where they are using readentity/writeentity. We should atleast create a common class, and then we can deprecate read/write entity later, for a new release. For now, I propose to make a backward compatible change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2702) listPartitionsByFilter only supports non-string partitions
[
https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach reassigned HIVE-2702:
Assignee: Aniket Mokashi
listPartitionsByFilter only supports non-string partitions
--
Key: HIVE-2702
URL: https://issues.apache.org/jira/browse/HIVE-2702
Project: Hive
Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Aniket Mokashi
Assignee: Aniket Mokashi
Attachments: HIVE-2702.1.patch, HIVE-2702.D2043.1.patch
listPartitionsByFilter supports only non-string partitions. This is because
its explicitly specified in generateJDOFilterOverPartitions in
ExpressionTree.java.
//Can only support partitions whose types are string
if( ! table.getPartitionKeys().get(partitionColumnIndex).
getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
throw new MetaException
(Filtering is supported only on partition keys of type string);
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2838) cleanup readentity/writeentity
[ https://issues.apache.org/jira/browse/HIVE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225441#comment-13225441 ] Phabricator commented on HIVE-2838: --- kevinwilfong has requested changes to the revision HIVE-2838 [jira] cleanup readentity/writeentity. The code looks good. Noticed a few relics left over from WriteEntity in the comments in Entity. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:30-31 Could you fix this comment. ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:37 This one too. ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:70 Could you split this into two. ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:133 Could you change written to to accessed or something like that. ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:152 here too. ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:180 here too. REVISION DETAIL https://reviews.facebook.net/D2193 BRANCH svn cleanup readentity/writeentity -- Key: HIVE-2838 URL: https://issues.apache.org/jira/browse/HIVE-2838 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: HIVE-2838.D2193.1.patch Ideally, there should be one common entity instead of readentity/writeentity. Unfortunately, that would be a backward incompatible change since users os hive might have written there own hooks, where they are using readentity/writeentity. We should atleast create a common class, and then we can deprecate read/write entity later, for a new release. For now, I propose to make a backward compatible change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Automatic/parallel patch testing
I won't be able to work on this right now. John, do you have some cycles ? On 3/7/12 9:30 PM, Amareshwari Sri Ramadasu amar...@yahoo-inc.com wrote: +1 for HIVE-1175 -Amareshwari On 3/8/12 3:21 AM, Ashutosh Chauhan hashut...@apache.org wrote: I am very much in favor of https://issues.apache.org/jira/browse/HIVE-1175 since then it publishes back on jira.. and everyone is on same page. I think Project VP has access to apache hudson. John / Namit, Can you set this up for Hive? Ashutosh On Wed, Mar 7, 2012 at 13:35, Edward Capriolo edlinuxg...@gmail.com wrote: I am trying to get myself more involved in the patch review and committing process, but running ant tests takes multiple hours. Two ideas: https://issues.apache.org/jira/browse/HIVE-1175 https://cwiki.apache.org/Hive/unit-test-parallel-execution.html Can we get a farm of test servers to get testing done faster what are other committers currently doing? I would not mind committing resources servers/$ to the cause Thanks, Edward
[jira] [Commented] (HIVE-2853) Add pre event listeners to metastore
[ https://issues.apache.org/jira/browse/HIVE-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225449#comment-13225449 ] Phabricator commented on HIVE-2853: --- njain has accepted the revision HIVE-2853 [jira] Add pre event listeners to metastore. REVISION DETAIL https://reviews.facebook.net/D2175 BRANCH svn Add pre event listeners to metastore Key: HIVE-2853 URL: https://issues.apache.org/jira/browse/HIVE-2853 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2853.D2175.1.patch Currently there are event listeners in the metastore which run after the completion of a method. It would be useful to have similar hooks which run before the metastore method is executed. These can be used to make validating names, locations, etc. customizable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225502#comment-13225502 ] Phabricator commented on HIVE-2646: --- cwsteinbach has requested changes to the revision HIVE-2646 [jira] Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs. INLINE COMMENTS build-common.xml:68 Please move this to build.properties build-common.xml:86 Is this specific to 0.23? If so please add a comment. build-common.xml:148 Since Hadoop 1.0.0 is now a reality, I think it would be good to always prefix any occurrences of version numbers with the major number, e.g. 0.20 instead of 20 build-common.xml:154 We should be consistent about the naming scheme in relation to versions, e.g. ivy-retrieve-hadoop-0.20, ivy-resolve-hadoop-0.20, hadoop-0.20-shim, etc. build.properties:20 Please add some comments explaining what this property does. testutils/hadoop:48 We should probably just insist that $HIVE_TEST_CLASSPATH is set and error out if it isn't. build.xml:700 This target is broken. build.xml:740 This target is broken. REVISION DETAIL https://reviews.facebook.net/D2133 BRANCH HIVE-2646-dev-branch Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Priority: Critical Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2646: - Status: Open (was: Patch Available) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Priority: Critical Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2838) cleanup readentity/writeentity
[ https://issues.apache.org/jira/browse/HIVE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2838: -- Attachment: HIVE-2838.D2193.2.patch njain updated the revision HIVE-2838 [jira] cleanup readentity/writeentity. Reviewers: JIRA, kevinwilfong Comments REVISION DETAIL https://reviews.facebook.net/D2193 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java cleanup readentity/writeentity -- Key: HIVE-2838 URL: https://issues.apache.org/jira/browse/HIVE-2838 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: HIVE-2838.D2193.1.patch, HIVE-2838.D2193.2.patch Ideally, there should be one common entity instead of readentity/writeentity. Unfortunately, that would be a backward incompatible change since users os hive might have written there own hooks, where they are using readentity/writeentity. We should atleast create a common class, and then we can deprecate read/write entity later, for a new release. For now, I propose to make a backward compatible change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2838) cleanup readentity/writeentity
[ https://issues.apache.org/jira/browse/HIVE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2838: - Status: Patch Available (was: Open) cleanup readentity/writeentity -- Key: HIVE-2838 URL: https://issues.apache.org/jira/browse/HIVE-2838 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: HIVE-2838.D2193.1.patch, HIVE-2838.D2193.2.patch Ideally, there should be one common entity instead of readentity/writeentity. Unfortunately, that would be a backward incompatible change since users os hive might have written there own hooks, where they are using readentity/writeentity. We should atleast create a common class, and then we can deprecate read/write entity later, for a new release. For now, I propose to make a backward compatible change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2853) Add pre event listeners to metastore
[ https://issues.apache.org/jira/browse/HIVE-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225533#comment-13225533 ] Ashutosh Chauhan commented on HIVE-2853: @Namit, Will it be ok for you to hold-on the commit before others get a chance to review this. Looks like a user facing feature is getting introduced here. Want to understand it a bit. Add pre event listeners to metastore Key: HIVE-2853 URL: https://issues.apache.org/jira/browse/HIVE-2853 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2853.D2175.1.patch Currently there are event listeners in the metastore which run after the completion of a method. It would be useful to have similar hooks which run before the metastore method is executed. These can be used to make validating names, locations, etc. customizable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2853) Add pre event listeners to metastore
[ https://issues.apache.org/jira/browse/HIVE-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225543#comment-13225543 ] Phabricator commented on HIVE-2853: --- kevinwilfong has commented on the revision HIVE-2853 [jira] Add pre event listeners to metastore. Spoke offline with Namit, I'm going to make the MetaStorePreEventListener methods into a single, more generic method, so that it will be possible to add more listeners to more metastore methods without breaking existing implementations of MetaStorePreEventListener. REVISION DETAIL https://reviews.facebook.net/D2175 BRANCH svn Add pre event listeners to metastore Key: HIVE-2853 URL: https://issues.apache.org/jira/browse/HIVE-2853 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2853.D2175.1.patch Currently there are event listeners in the metastore which run after the completion of a method. It would be useful to have similar hooks which run before the metastore method is executed. These can be used to make validating names, locations, etc. customizable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225579#comment-13225579 ] Phabricator commented on HIVE-2646: --- cwsteinbach has commented on the revision HIVE-2646 [jira] Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs. INLINE COMMENTS build.xml:542 This patch breaks the eclipse classpath template. Can you please investigate the feasibility of intregrating IvyDE into the build? HIVE-2739 covers that task, but it would be great to get this all done in one patch. Note that ZK recently added support for IvyDE in ZOOKEEPER-1178, so you may be able to look at that patch for hints. REVISION DETAIL https://reviews.facebook.net/D2133 BRANCH HIVE-2646-dev-branch Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Priority: Critical Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225581#comment-13225581 ] Phabricator commented on HIVE-2646: --- thw has commented on the revision HIVE-2646 [jira] Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs. INLINE COMMENTS shims/ivy.xml:34 Can we make separate ivy files for 0.20 and 0.23 and include dynamically? This would eliminate the need for the hadoopXXX configuration names. build-common.xml:154 Please make a single ivy-retrieve-hadoop target or macro and parameterize the shim name. REVISION DETAIL https://reviews.facebook.net/D2133 BRANCH HIVE-2646-dev-branch Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Priority: Critical Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2853) Add pre event listeners to metastore
[ https://issues.apache.org/jira/browse/HIVE-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225588#comment-13225588 ] Kevin Wilfong commented on HIVE-2853: - @Ashutosh This feature is just a new type of hook, unless you provide an implementation (and I am not providing any implementations in this patch) it should have no effect, and in particular, users will not see any change. Add pre event listeners to metastore Key: HIVE-2853 URL: https://issues.apache.org/jira/browse/HIVE-2853 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2853.D2175.1.patch Currently there are event listeners in the metastore which run after the completion of a method. It would be useful to have similar hooks which run before the metastore method is executed. These can be used to make validating names, locations, etc. customizable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2813) Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.
[ https://issues.apache.org/jira/browse/HIVE-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225609#comment-13225609 ] Phabricator commented on HIVE-2813: --- kevinwilfong has accepted the revision HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.. +1 Looks good to me now. @Carl It looks like he's addressed all of your concerns. I'll commit on Monday unless you see any other issues. REVISION DETAIL https://reviews.facebook.net/D1803 BRANCH trunk Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match. -- Key: HIVE-2813 URL: https://issues.apache.org/jira/browse/HIVE-2813 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.8.1, 0.9.0 Reporter: Dmitry Soshnikov Priority: Minor Labels: newbie, patch Fix For: 0.9.0 Attachments: HIVE-2813.D1803.2.patch, HIVE-2813.D1803.3.patch, HIVE-2813.D1803.4.patch, HIVE-2813.D1803.5.patch Original Estimate: 24h Remaining Estimate: 24h Oftentimes people try to write queries like SELECT * FROM table WHERE ds = 2011-08-03 This won't work because quotes are missing, and it'll actually try to filter on ds = 2000. People run into this pretty regularly; a simple check on whether the types match exactly in partition predicates would make this a lot less likely to happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2783) Predicate Pushdown failure a view with union when using MapReduce2
[ https://issues.apache.org/jira/browse/HIVE-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenxiao Luo resolved HIVE-2783. Resolution: Fixed Fixed in MAPREDUCE-3952 Predicate Pushdown failure a view with union when using MapReduce2 -- Key: HIVE-2783 URL: https://issues.apache.org/jira/browse/HIVE-2783 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo ppd_union_view failure: [junit] Begin query: ppd_union_view.q [junit] 12/01/23 16:57:30 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:57:30 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:57:37 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:57:37 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:57:44 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:57:44 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:57:51 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:57:51 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:57:57 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:57:57 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:04 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:04 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:11 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:11 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:16 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:16 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:21 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:21 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:26 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:26 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:31 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:31 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:35 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:35 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:41 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:41 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:46 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:46 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:50 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:50 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:58:55 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:58:55 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:00 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:00 WARN
[jira] [Commented] (HIVE-2838) cleanup readentity/writeentity
[ https://issues.apache.org/jira/browse/HIVE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225622#comment-13225622 ] Phabricator commented on HIVE-2838: --- kevinwilfong has accepted the revision HIVE-2838 [jira] cleanup readentity/writeentity. +1 Will commit after tests pass. REVISION DETAIL https://reviews.facebook.net/D2193 BRANCH svn cleanup readentity/writeentity -- Key: HIVE-2838 URL: https://issues.apache.org/jira/browse/HIVE-2838 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: HIVE-2838.D2193.1.patch, HIVE-2838.D2193.2.patch Ideally, there should be one common entity instead of readentity/writeentity. Unfortunately, that would be a backward incompatible change since users os hive might have written there own hooks, where they are using readentity/writeentity. We should atleast create a common class, and then we can deprecate read/write entity later, for a new release. For now, I propose to make a backward compatible change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225629#comment-13225629 ] Phabricator commented on HIVE-2646: --- cwsteinbach has commented on the revision HIVE-2646 [jira] Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs. INLINE COMMENTS shims/ivy.xml:34 Aren't configurations the designated mechanism for handling situations like this in Ivy? I don't see how breaking this into multiple files will improve the situation. Won't that mean that we'll have to update multiple files every time a new dependency is added instead of just one? REVISION DETAIL https://reviews.facebook.net/D2133 BRANCH HIVE-2646-dev-branch Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Priority: Critical Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[
https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225635#comment-13225635
]
Phabricator commented on HIVE-2646:
---
abayer has commented on the revision HIVE-2646 [jira] Hive Ivy dependencies on
Hadoop should depend on jars directly, not tarballs.
I'll rework the ivy-{retrieve,resolve}-hadoopX targets get generalized - that
should be easy. And I agree - the configuration switch is the right way to
handle this.
I'll take a look at IvyDE.
REVISION DETAIL
https://reviews.facebook.net/D2133
BRANCH
HIVE-2646-dev-branch
Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
Key: HIVE-2646
URL: https://issues.apache.org/jira/browse/HIVE-2646
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Affects Versions: 0.8.0
Reporter: Andrew Bayer
Priority: Critical
Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.2.patch,
HIVE-2646.D2133.3.patch, HIVE-2646.diff.txt
The current Hive Ivy dependency logic for its Hadoop dependencies is
problematic - depending on the tarball and extracting the jars from there,
rather than depending on the jars directly. It'd be great if this was fixed
to actually have the jar dependencies defined directly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225646#comment-13225646 ] Phabricator commented on HIVE-2646: --- thw has commented on the revision HIVE-2646 [jira] Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs. INLINE COMMENTS shims/ivy.xml:34 My comment refers only to the hadoop dependencies below. You will never have hadoop23 and hadoop20 configuration at the same time. Below is what we did in Pig, but it gets really cluttered because now you have the stuff for 0.20 and 0.23, which is mutually exclusive, in one file. REVISION DETAIL https://reviews.facebook.net/D2133 BRANCH HIVE-2646-dev-branch Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Priority: Critical Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225665#comment-13225665 ] Phabricator commented on HIVE-2646: --- cwsteinbach has commented on the revision HIVE-2646 [jira] Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs. INLINE COMMENTS shims/ivy.xml:34 Seems like we can get the same benefits by ordering the dependencies listed in each ivy.xml file based on configuration. @Andrew: Can you look into doing this? Thanks. REVISION DETAIL https://reviews.facebook.net/D2133 BRANCH HIVE-2646-dev-branch Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs Key: HIVE-2646 URL: https://issues.apache.org/jira/browse/HIVE-2646 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.8.0 Reporter: Andrew Bayer Priority: Critical Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.diff.txt The current Hive Ivy dependency logic for its Hadoop dependencies is problematic - depending on the tarball and extracting the jars from there, rather than depending on the jars directly. It'd be great if this was fixed to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225683#comment-13225683 ] Phabricator commented on HIVE-2764: --- enis has commented on the revision HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup. Some comments about the patch: Changing the StorageHandler.configureJobProperties() to be called with the actual job is kind of intrusive, so I have given up on that approach, and go with the plan of obtaining the delegation tokens from the IF.getSplits() and OF.checkOutputSpecs() time. However, one problem with this approach is that although on the input side it works fine, on the output side, Hive does not use OF's properly. So, this patch adds a HiveOutputFormatImpl class, and changes the HOF to extend OF. The actual usage of OFs are not changed as a part of this patch, since it is out of the scope of this issue. However, not the HOFImpl class delgates the checkOutputSpecs() call to the underlying OutputFormat, so that if the OF wants to do any initialization, it can. REVISION DETAIL https://reviews.facebook.net/D2205 Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2764.D2205.1.patch, HIVE-2764_v0.patch As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2186) Dynamic Partitioning Failing because of characters not supported globStatus
[ https://issues.apache.org/jira/browse/HIVE-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225715#comment-13225715 ] Zhenxiao Luo commented on HIVE-2186: Seems '^' also needs to be escaped. If add: SELECT count(*) from escape_raw; SELECT count(*) from escape1; into escape1.q escape1 only get 127 rows, while, escape_raw get 128 rows(note that escapetest.txt has 128 rows). one row missing in escape1, which is the '^'. This is to be fixed in HIVE-2856. Comments are appreciated. Thanks, Zhenxiao Dynamic Partitioning Failing because of characters not supported globStatus --- Key: HIVE-2186 URL: https://issues.apache.org/jira/browse/HIVE-2186 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Siying Dong Assignee: Franklin Hu Fix For: 0.8.0 Attachments: hive-2186.1.patch, hive-2186.2.patch, hive-2186.3.patch, hive-2186.4.patch, hive-2186.5.patch Some dynamic queries failed on the stage of loading partitions if dynamic partition columns contain special characters. We need to escape all of them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2856) When integrating into MapReduce2, additional '^' in escape test
[ https://issues.apache.org/jira/browse/HIVE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225722#comment-13225722 ] Zhenxiao Luo commented on HIVE-2856: In testcase escape1.q: Seems '^' also needs to be escaped. If add: SELECT count from escape_raw; SELECT count from escape1; into escape1.q escape1 only get 127 rows, while, escape_raw get 128 rows(note that escapetest.txt has 128 rows). one row missing in escape1, which is the '^'. When integrating into MapReduce2, additional '^' in escape test --- Key: HIVE-2856 URL: https://issues.apache.org/jira/browse/HIVE-2856 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Additional '^' in escape test: [junit] Begin query: escape1.q [junit] Copying file: file:/home/cloudera/Code/hive/data/files/escapetest.txt [junit] 12/01/23 15:22:15 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 15:22:15 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]*_[0-9]* -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/escape1.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/escape1.q.out [junit] 893d892 [junit] 1 1 ^ [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1(TestCliDriver.java:131) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) [junit] at junit.framework.TestSuite.run(TestSuite.java:238) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] Exception: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2837) insert into external tables should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2837: -- Attachment: HIVE-2837.D2211.1.patch njain requested code review of HIVE-2837 [jira] insert into external tables should not be allowed. Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2837 HIVE-2837 By default, the behavior does not change. It is backward compatible This is a very risky thing to allow. Since, the external tables can point to any user location, which can potentially corrupt some other tables. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2211 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/insertexternal1.q.out ql/src/test/queries/clientnegative/insertexternal1.q ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4845/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. insert into external tables should not be allowed - Key: HIVE-2837 URL: https://issues.apache.org/jira/browse/HIVE-2837 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: HIVE-2837.D2211.1.patch This is a very risky thing to allow. Since, the external tables can point to any user location, which can potentially corrupt some other tables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1634) Allow access to Primitive types stored in binary format in HBase
[ https://issues.apache.org/jira/browse/HIVE-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225744#comment-13225744 ] Ashutosh Chauhan commented on HIVE-1634: Committed to trunk. Thanks, Carl for the review. Credit for this goes to Basab Maulik who did initial patch. I just rebased and took care of the comments by reviewers. Thanks, Basab! Allow access to Primitive types stored in binary format in HBase Key: HIVE-1634 URL: https://issues.apache.org/jira/browse/HIVE-1634 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.7.0, 0.8.0, 0.9.0 Reporter: Basab Maulik Assignee: Ashutosh Chauhan Fix For: 0.9.0 Attachments: HIVE-1634.0.patch, HIVE-1634.1.patch, HIVE-1634.D1581.1.patch, HIVE-1634.D1581.2.patch, HIVE-1634.D1581.3.patch, TestHiveHBaseExternalTable.java, hive-1634_3.patch This addresses HIVE-1245 in part, for atomic or primitive types. The serde property hbase.columns.storage.types = -,b,b,b,b,b,b,b,b is a specification of the storage option for the corresponding column in the serde property hbase.columns.mapping. Allowed values are '-' for table default, 's' for standard string storage, and 'b' for binary storage as would be obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families use a colon separated pair such as 's:b' for the key and value part specifiers respectively. See the test cases and queries for HBase handler for additional examples. There is also a table property hbase.table.default.storage.type = string to specify a table level default storage type. The other valid specification is binary. The table level default is overridden by a column level specification. This control is available for the boolean, tinyint, smallint, int, bigint, float, and double primitive types. The attached patch also relaxes the mapping of map types to HBase column families to allow any primitive type to be the map key. Attached is a program for creating a table and populating it in HBase. The external table in Hive can access the data as shown in the example below. hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties (hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.691 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 NULLNULLNULLNULLNULLTest-String NULLNULL Time taken: 0.346 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.139 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,b,b,b ) tblproperties ( hbase.table.name = TestHiveHBaseExternalTable, hbase.table.default.storage.type = string); OK Time taken: 0.139 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291 Time taken: 0.151 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.154 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,-,b,b ) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.347 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291 Time taken: 0.245 seconds hive -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Updated] (HIVE-1634) Allow access to Primitive types stored in binary format in HBase
[ https://issues.apache.org/jira/browse/HIVE-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-1634: --- Resolution: Fixed Status: Resolved (was: Patch Available) Allow access to Primitive types stored in binary format in HBase Key: HIVE-1634 URL: https://issues.apache.org/jira/browse/HIVE-1634 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.7.0, 0.8.0, 0.9.0 Reporter: Basab Maulik Assignee: Ashutosh Chauhan Fix For: 0.9.0 Attachments: HIVE-1634.0.patch, HIVE-1634.1.patch, HIVE-1634.D1581.1.patch, HIVE-1634.D1581.2.patch, HIVE-1634.D1581.3.patch, TestHiveHBaseExternalTable.java, hive-1634_3.patch This addresses HIVE-1245 in part, for atomic or primitive types. The serde property hbase.columns.storage.types = -,b,b,b,b,b,b,b,b is a specification of the storage option for the corresponding column in the serde property hbase.columns.mapping. Allowed values are '-' for table default, 's' for standard string storage, and 'b' for binary storage as would be obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families use a colon separated pair such as 's:b' for the key and value part specifiers respectively. See the test cases and queries for HBase handler for additional examples. There is also a table property hbase.table.default.storage.type = string to specify a table level default storage type. The other valid specification is binary. The table level default is overridden by a column level specification. This control is available for the boolean, tinyint, smallint, int, bigint, float, and double primitive types. The attached patch also relaxes the mapping of map types to HBase column families to allow any primitive type to be the map key. Attached is a program for creating a table and populating it in HBase. The external table in Hive can access the data as shown in the example below. hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties (hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.691 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 NULLNULLNULLNULLNULLTest-String NULLNULL Time taken: 0.346 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.139 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,b,b,b ) tblproperties ( hbase.table.name = TestHiveHBaseExternalTable, hbase.table.default.storage.type = string); OK Time taken: 0.139 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291 Time taken: 0.151 seconds hive drop table TestHiveHBaseExternalTable; OK Time taken: 0.154 seconds hive create external table TestHiveHBaseExternalTable (key string, c_bool boolean, c_byte tinyint, c_short smallint, c_int int, c_long bigint, c_string string, c_float float, c_double double) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( hbase.columns.mapping = :key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double, hbase.columns.storage.types = -,b,b,b,b,b,-,b,b ) tblproperties (hbase.table.name = TestHiveHBaseExternalTable); OK Time taken: 0.347 seconds hive select * from TestHiveHBaseExternalTable; OK key-1 true-128-32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291 Time taken: 0.245 seconds hive -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2837) insert into external tables should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225750#comment-13225750 ] Phabricator commented on HIVE-2837: --- kevinwilfong has added CCs to the revision HIVE-2837 [jira] insert into external tables should not be allowed. Added CCs: kevinwilfong REVISION DETAIL https://reviews.facebook.net/D2211 insert into external tables should not be allowed - Key: HIVE-2837 URL: https://issues.apache.org/jira/browse/HIVE-2837 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: HIVE-2837.D2211.1.patch This is a very risky thing to allow. Since, the external tables can point to any user location, which can potentially corrupt some other tables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2805) Move metastore upgrade scripts labeled 0.10.0 into scripts labeled 0.9.0
[ https://issues.apache.org/jira/browse/HIVE-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2805: Status: Patch Available (was: Open) Move metastore upgrade scripts labeled 0.10.0 into scripts labeled 0.9.0 Key: HIVE-2805 URL: https://issues.apache.org/jira/browse/HIVE-2805 Project: Hive Issue Type: Task Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2805.D1743.1.patch Move contents of upgrade-0.9.0-to-0.10.0.mysql.sql, upgrade-0.9.0-to-0.10.0.derby.sql into upgrade-0.8.0-to-0.9.0.mysql.sql, upgrade-0.8.0-to-0.9.0.derby.sql Rename hive-schema-0.10.0.derby.sql, hive-schema-0.10.0.mysql.sql to hive-schema-0.9.0.derby.sql, hive-schema-0.9.0.mysql.sql -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2837) insert into external tables should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2837: -- Attachment: HIVE-2837.D2211.2.patch njain updated the revision HIVE-2837 [jira] insert into external tables should not be allowed. Reviewers: JIRA Change error message REVISION DETAIL https://reviews.facebook.net/D2211 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/insertexternal1.q.out ql/src/test/queries/clientnegative/insertexternal1.q ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java insert into external tables should not be allowed - Key: HIVE-2837 URL: https://issues.apache.org/jira/browse/HIVE-2837 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: HIVE-2837.D2211.1.patch, HIVE-2837.D2211.2.patch This is a very risky thing to allow. Since, the external tables can point to any user location, which can potentially corrupt some other tables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2186) Dynamic Partitioning Failing because of characters not supported globStatus
[ https://issues.apache.org/jira/browse/HIVE-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225761#comment-13225761 ] Zhenxiao Luo commented on HIVE-2186: Whoops, seems escape1.q.out is loaded as data, not ascii files. It is difficult to check diffs for new patches. Dynamic Partitioning Failing because of characters not supported globStatus --- Key: HIVE-2186 URL: https://issues.apache.org/jira/browse/HIVE-2186 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Siying Dong Assignee: Franklin Hu Fix For: 0.8.0 Attachments: hive-2186.1.patch, hive-2186.2.patch, hive-2186.3.patch, hive-2186.4.patch, hive-2186.5.patch Some dynamic queries failed on the stage of loading partitions if dynamic partition columns contain special characters. We need to escape all of them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2856) When integrating into MapReduce2, additional '^' in escape test
[ https://issues.apache.org/jira/browse/HIVE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenxiao Luo updated HIVE-2856: --- Attachment: HIVE-2856.1.patch.txt 1. '^' should be escaped 2. the previous escape1.q.out is incorrect, the '^' row is always missing in table escape1. table escape1 has one less row than escape_raw. 3. #1 and #2 together leads to testcase escape1 always passing 4. for some reason, escape1.q.out is loaded as binary file, not ascii file. git diff could not get diffs for binary file 5. This patch only includes code changes and escape1.q change. escape1.q.out not included(could not get via git diff) When integrating into MapReduce2, additional '^' in escape test --- Key: HIVE-2856 URL: https://issues.apache.org/jira/browse/HIVE-2856 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Attachments: HIVE-2856.1.patch.txt Additional '^' in escape test: [junit] Begin query: escape1.q [junit] Copying file: file:/home/cloudera/Code/hive/data/files/escapetest.txt [junit] 12/01/23 15:22:15 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 15:22:15 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]*_[0-9]* -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/escape1.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/escape1.q.out [junit] 893d892 [junit] 1 1 ^ [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1(TestCliDriver.java:131) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) [junit] at junit.framework.TestSuite.run(TestSuite.java:238) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] Exception: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. [junit] See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2857) QTestUtil.cleanUp() fails with FileNotException on 0.23
QTestUtil.cleanUp() fails with FileNotException on 0.23 --- Key: HIVE-2857 URL: https://issues.apache.org/jira/browse/HIVE-2857 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2857) QTestUtil.cleanUp() fails with FileNotException on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225808#comment-13225808 ] Carl Steinbach commented on HIVE-2857: -- On 0.23 FileSystem.listStatus() throws FileNotFoundException instead of returning null. This causes the MinimrCliDriver tests to fail since QTestUtil.cleanUp is called before the warehouse directory is created. QTestUtil.cleanUp() fails with FileNotException on 0.23 --- Key: HIVE-2857 URL: https://issues.apache.org/jira/browse/HIVE-2857 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2837) insert into external tables should not be allowed
[ https://issues.apache.org/jira/browse/HIVE-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225809#comment-13225809 ] Phabricator commented on HIVE-2837: --- kevinwilfong has accepted the revision HIVE-2837 [jira] insert into external tables should not be allowed. +1 Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D2211 BRANCH svn insert into external tables should not be allowed - Key: HIVE-2837 URL: https://issues.apache.org/jira/browse/HIVE-2837 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: HIVE-2837.D2211.1.patch, HIVE-2837.D2211.2.patch This is a very risky thing to allow. Since, the external tables can point to any user location, which can potentially corrupt some other tables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2858) Cache remote map reduce job stack traces for additional logging
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2858: Status: Patch Available (was: Open) Cache remote map reduce job stack traces for additional logging --- Key: HIVE-2858 URL: https://issues.apache.org/jira/browse/HIVE-2858 Project: Hive Issue Type: Improvement Components: Logging Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2858.D2223.1.patch Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2858) Cache remote map reduce job stack traces for additional logging
[ https://issues.apache.org/jira/browse/HIVE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2858: -- Attachment: HIVE-2858.D2223.1.patch kevinwilfong requested code review of HIVE-2858 [jira] Cache remote map reduce job stack traces for additional logging. Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2858 Modified the JobDebugger and TaskLogProcessor to parse out stack traces from failed task logs. These are stored in a mapping from job ID to stack traces in the SessionState. Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2223 AFFECTED FILES conf/hive-default.xml.template build-common.xml common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateStackTracesHook.java ql/src/test/queries/clientnegative/mapreduce_stack_trace.q ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4875/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. Cache remote map reduce job stack traces for additional logging --- Key: HIVE-2858 URL: https://issues.apache.org/jira/browse/HIVE-2858 Project: Hive Issue Type: Improvement Components: Logging Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2858.D2223.1.patch Currently we are parsing the task logs for failed jobs for information to display to the user in the CLI. In addition, we could parse those logs for stack traces and store e them in the SessionState. This way, when we log failed queries, these will give us a decent idea of why those queries failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2838) cleanup readentity/writeentity
[ https://issues.apache.org/jira/browse/HIVE-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2838: Resolution: Fixed Fix Version/s: 0.9.0 Status: Resolved (was: Patch Available) Committed, thanks Namit. cleanup readentity/writeentity -- Key: HIVE-2838 URL: https://issues.apache.org/jira/browse/HIVE-2838 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.9.0 Attachments: HIVE-2838.D2193.1.patch, HIVE-2838.D2193.2.patch Ideally, there should be one common entity instead of readentity/writeentity. Unfortunately, that would be a backward incompatible change since users os hive might have written there own hooks, where they are using readentity/writeentity. We should atleast create a common class, and then we can deprecate read/write entity later, for a new release. For now, I propose to make a backward compatible change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2857) QTestUtil.cleanUp() fails with FileNotException on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2857: -- Attachment: HIVE-2857.D2229.1.patch cwsteinbach requested code review of HIVE-2857 [jira] QTestUtil.cleanUp() fails with FileNotException on 0.23. Reviewers: JIRA HIVE-2857. QTestUtil.cleanUp() fails with FileNotException on 0.23 TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2229 AFFECTED FILES ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4881/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. QTestUtil.cleanUp() fails with FileNotException on 0.23 --- Key: HIVE-2857 URL: https://issues.apache.org/jira/browse/HIVE-2857 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.9.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 Attachments: HIVE-2857.D2229.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2832) Cache error messages for additional logging
[ https://issues.apache.org/jira/browse/HIVE-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225867#comment-13225867 ] Phabricator commented on HIVE-2832: --- njain has accepted the revision HIVE-2832 [jira] Cache error messages for additional logging. REVISION DETAIL https://reviews.facebook.net/D2025 BRANCH svn Cache error messages for additional logging --- Key: HIVE-2832 URL: https://issues.apache.org/jira/browse/HIVE-2832 Project: Hive Issue Type: Improvement Components: Logging Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2832.D2025.1.patch, HIVE-2832.D2025.2.patch It would be good if we could cache logs written to SessionState.err so that they could be exposed to hooks for additional logging. This would allow logging of error messages with the queries that failed in a central location. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2832) Cache error messages for additional logging
[ https://issues.apache.org/jira/browse/HIVE-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225868#comment-13225868 ] Phabricator commented on HIVE-2832: --- njain has commented on the revision HIVE-2832 [jira] Cache error messages for additional logging. +1 REVISION DETAIL https://reviews.facebook.net/D2025 Cache error messages for additional logging --- Key: HIVE-2832 URL: https://issues.apache.org/jira/browse/HIVE-2832 Project: Hive Issue Type: Improvement Components: Logging Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2832.D2025.1.patch, HIVE-2832.D2025.2.patch It would be good if we could cache logs written to SessionState.err so that they could be exposed to hooks for additional logging. This would allow logging of error messages with the queries that failed in a central location. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1299 - Failure
Changes for Build #1299 [hashutosh] HIVE-1634: Allow access to Primitive types stored in binary format in HBase (Basab Maulik, Ashutosh Chauhan via hashutosh) 1 tests failed. REGRESSION: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try ant test ... -Dtest.silent=false to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:10262) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1299) Status: Failure Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1299/ to view the results.
[jira] [Commented] (HIVE-1634) Allow access to Primitive types stored in binary format in HBase
[ https://issues.apache.org/jira/browse/HIVE-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225871#comment-13225871 ] Hudson commented on HIVE-1634: -- Integrated in Hive-trunk-h0.21 #1299 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1299/]) HIVE-1634: Allow access to Primitive types stored in binary format in HBase (Basab Maulik, Ashutosh Chauhan via hashutosh) (Revision 1298673) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1298673 Files : * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsPublisher.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java * /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java * /hive/trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestSetup.java * /hive/trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java * /hive/trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java * /hive/trunk/hbase-handler/src/test/queries/hbase_binary_external_table_queries.q * /hive/trunk/hbase-handler/src/test/queries/hbase_binary_map_queries.q * /hive/trunk/hbase-handler/src/test/queries/hbase_binary_storage_queries.q * /hive/trunk/hbase-handler/src/test/results/hbase_binary_external_table_queries.q.out * /hive/trunk/hbase-handler/src/test/results/hbase_binary_map_queries.q.out * /hive/trunk/hbase-handler/src/test/results/hbase_binary_storage_queries.q.out * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObject.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyPrimitive.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazydio * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazydio/LazyDioBoolean.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazydio/LazyDioByte.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazydio/LazyDioDouble.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazydio/LazyDioFloat.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazydio/LazyDioInteger.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazydio/LazyDioLong.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazydio/LazyDioShort.java Allow access to Primitive types stored in binary format in HBase Key: HIVE-1634 URL: https://issues.apache.org/jira/browse/HIVE-1634 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.7.0, 0.8.0, 0.9.0 Reporter: Basab Maulik Assignee: Ashutosh Chauhan Fix For: 0.9.0 Attachments: HIVE-1634.0.patch, HIVE-1634.1.patch, HIVE-1634.D1581.1.patch, HIVE-1634.D1581.2.patch, HIVE-1634.D1581.3.patch, TestHiveHBaseExternalTable.java, hive-1634_3.patch This addresses HIVE-1245 in part, for atomic or primitive types. The serde property hbase.columns.storage.types = -,b,b,b,b,b,b,b,b is a specification of the storage option for the corresponding column in the serde property hbase.columns.mapping. Allowed values are '-' for table default, 's' for standard string storage, and 'b' for binary storage as would be obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families use a colon separated pair such as 's:b' for the key and value part specifiers respectively. See the test cases and queries for HBase handler for additional examples. There is also a table property hbase.table.default.storage.type = string to specify a table level default storage type. The other valid specification is binary. The table level default is overridden by a column level specification. This control is available for the boolean, tinyint, smallint, int, bigint, float, and double primitive types. The attached patch also relaxes the mapping of map types to HBase column families to allow any primitive type to be the map key. Attached is a program for creating a table and populating it in HBase. The external table in Hive can access the data as shown in the
