[jira] [Commented] (HIVE-2504) Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory
[ https://issues.apache.org/jira/browse/HIVE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214629#comment-13214629 ] Sho Shimauchi commented on HIVE-2504: - sure, I'll file a JIRA and submit a patch. > Warehouse table subdirectories should inherit the group permissions of the > warehouse parent directory > - > > Key: HIVE-2504 > URL: https://issues.apache.org/jira/browse/HIVE-2504 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach >Assignee: Chinna Rao Lalam > Fix For: 0.9.0 > > Attachments: HIVE-2504.patch > > > When the Hive Metastore creates a subdirectory in the Hive warehouse for > a new table it does so with the default HDFS permissions. Since the default > dfs.umask value is 022, this means that the new subdirectory will not inherit > the > group write permissions of the hive warehouse directory. > We should make the umask used by Warehouse.mkdirs() configurable, and set > it to use a default value of 002. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2803) utc_from_timestamp and utc_to_timestamp returns incorrect results.
[
https://issues.apache.org/jira/browse/HIVE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tamtam180 updated HIVE-2803:
Status: Patch Available (was: Open)
> utc_from_timestamp and utc_to_timestamp returns incorrect results.
> --
>
> Key: HIVE-2803
> URL: https://issues.apache.org/jira/browse/HIVE-2803
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 0.8.0
>Reporter: tamtam180
> Attachments: HIVE-2803.1.patch.txt
>
>
> How to reproduce:
> {noformat}
> $ echo "2011-12-25 09:00:00.123456" > /tmp/data5.txt
> hive> create table ts1(t1 timestamp);
> hive> load data local inpath '/tmp/data5.txt' overwrite into table ts1;
> hive> select t1, from_utc_timestamp(t1, 'JST'), from_utc_timestamp(t1, 'JST')
> from ts1 limit 1;
> {noformat}
> The following result is expected:
> {noformat}
> 2011-12-25 09:00:00.123456 2011-12-25 18:00:00.123456 2011-12-25
> 18:00:00.123456
> {noformat}
> However, the above query return incorrect result like this:
> {noformat}
> 2011-12-26 03:00:00.492456 2011-12-26 03:00:00.492456 2011-12-26
> 03:00:00.492456
> {noformat}
> This is because GenericUDFFromUtcTimestamp.applyOffset() does setTime()
> improperly.
> On evaluating query, timestamp argument always returns the same instance.
> GenericUDFFromUtcTimestamp.applyOffset() does setTime() on the instance.
> That means it adds all offsets in the query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2773) HiveStorageHandler.configureTableJobProperites() should let the handler know wether it is configuration for input or output
[ https://issues.apache.org/jira/browse/HIVE-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214835#comment-13214835 ] Francis Liu commented on HIVE-2773: --- Since it's an interface, things will still break not unless they extended DefaultStorageHandler. In any case will ask user@hive. > HiveStorageHandler.configureTableJobProperites() should let the handler know > wether it is configuration for input or output > --- > > Key: HIVE-2773 > URL: https://issues.apache.org/jira/browse/HIVE-2773 > Project: Hive > Issue Type: Improvement >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hcatalog, storage_handler > Attachments: HIVE-2773.D1815.1.patch, HIVE-2773.patch > > > HiveStorageHandler.configureTableJobProperties() is called to allow the > storage handler to setup any properties that the underlying > inputformat/outputformat/serde may need. But the handler implementation does > not know whether it is being called for configuring input or output. This > makes it a problem for handlers which sets an external state. In the case of > HCatalog's HBase storageHandler, whenever a write needs to be configured we > create a write transaction which needs to be committed or aborted later on. > In this case configuring for both input and output each time > configureTableJobProperties() is called would not be desirable. This has > become an issue since HCatalog is dropping storageDrivers for SerDe and > StorageHandler (see HCATALOG-237). > My proposal is to replace configureTableJobProperties() with two methods: > configureInputJobProperties() > configureOutputJobProperties() > Each method will have the same signature. I cursory look at the code and I > believe changes should be straighforward also given that we are not really > changing anything just splitting responsibility. If the community is fine > with this approach I will go ahead and create a aptch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2781) HBaseSerDe should allow users to specify the timestamp passed to Puts
[ https://issues.apache.org/jira/browse/HIVE-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214836#comment-13214836 ] Francis Liu commented on HIVE-2781: --- You submitted a patch first. I just wanted to point out that your implementation was not what I had in mind when I filed this Jira. > HBaseSerDe should allow users to specify the timestamp passed to Puts > -- > > Key: HIVE-2781 > URL: https://issues.apache.org/jira/browse/HIVE-2781 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Francis Liu >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2781.D1863.1.patch, HIVE-2781.D1863.2.patch, > HIVE-2781.D1881.1.patch > > > Users may want to specify the timestamp used for Put requests to hbase. Thus > enabling users to have the same timestamp for a single batch of writes. Which > would be useful for a number of things. HCatalog's HBase storageHandler > implementation makes use of this feature to provide users with snapshot > isolation and write transactions. My proposal is to add the timestamp option > as a final static member: > public static final long HBASE_PUT_TIMESTAMP = "hbase.put_timestamp" > And passing this value to all the Puts created by serialize() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2773) HiveStorageHandler.configureTableJobProperites() should let the handler know wether it is configuration for input or output
[ https://issues.apache.org/jira/browse/HIVE-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214854#comment-13214854 ] Alexander Lorenz-Alten commented on HIVE-2773: -- Since Hive need the new methods to work with HCatalog the interface should be extended. Later it can be dropped or marked as obsolete. For a flumeNG hive sink the new methods are a good improvement. > HiveStorageHandler.configureTableJobProperites() should let the handler know > wether it is configuration for input or output > --- > > Key: HIVE-2773 > URL: https://issues.apache.org/jira/browse/HIVE-2773 > Project: Hive > Issue Type: Improvement >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hcatalog, storage_handler > Attachments: HIVE-2773.D1815.1.patch, HIVE-2773.patch > > > HiveStorageHandler.configureTableJobProperties() is called to allow the > storage handler to setup any properties that the underlying > inputformat/outputformat/serde may need. But the handler implementation does > not know whether it is being called for configuring input or output. This > makes it a problem for handlers which sets an external state. In the case of > HCatalog's HBase storageHandler, whenever a write needs to be configured we > create a write transaction which needs to be committed or aborted later on. > In this case configuring for both input and output each time > configureTableJobProperties() is called would not be desirable. This has > become an issue since HCatalog is dropping storageDrivers for SerDe and > StorageHandler (see HCATALOG-237). > My proposal is to replace configureTableJobProperties() with two methods: > configureInputJobProperties() > configureOutputJobProperties() > Each method will have the same signature. I cursory look at the code and I > believe changes should be straighforward also given that we are not really > changing anything just splitting responsibility. If the community is fine > with this approach I will go ahead and create a aptch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2490) Add reset operation and average time attribute to Metrics MBean.
[ https://issues.apache.org/jira/browse/HIVE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2490: Attachment: HIVE-2490.1.patch.txt > Add reset operation and average time attribute to Metrics MBean. > > > Key: HIVE-2490 > URL: https://issues.apache.org/jira/browse/HIVE-2490 > Project: Hive > Issue Type: New Feature >Reporter: Kevin Wilfong >Assignee: Kevin Wilfong > Attachments: HIVE-2490.1.patch.txt > > > We should add a reset operation to the Metrics MBean, which will set all the > counters to 0. > Note: Deleting the counters from the map of attributes was not suggested > because that could break any scripts that get the list of attributes from the > bean and then the values of each attribute. Also, 0 is unlikely to be an > actual value for any counter, and it will not break the increment > functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2490) Add reset operation and average time attribute to Metrics MBean.
[ https://issues.apache.org/jira/browse/HIVE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2490: Status: Patch Available (was: Open) > Add reset operation and average time attribute to Metrics MBean. > > > Key: HIVE-2490 > URL: https://issues.apache.org/jira/browse/HIVE-2490 > Project: Hive > Issue Type: New Feature >Reporter: Kevin Wilfong >Assignee: Kevin Wilfong > Attachments: HIVE-2490.1.patch.txt > > > We should add a reset operation to the Metrics MBean, which will set all the > counters to 0. > Note: Deleting the counters from the map of attributes was not suggested > because that could break any scripts that get the list of attributes from the > bean and then the values of each attribute. Also, 0 is unlikely to be an > actual value for any counter, and it will not break the increment > functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2816) Extend IMPORT/EXPORT to Support Metata Only Operations
Extend IMPORT/EXPORT to Support Metata Only Operations -- Key: HIVE-2816 URL: https://issues.apache.org/jira/browse/HIVE-2816 Project: Hive Issue Type: New Feature Components: Import/Export Reporter: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2816) Extend IMPORT/EXPORT to Support Metata Only Operations
[ https://issues.apache.org/jira/browse/HIVE-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214914#comment-13214914 ] Carl Steinbach commented on HIVE-2816: -- IMPORT/EXPORT commands were added in HIVE-1918. This ticket encompasses the task of extending this feature to support metadata-only IMPORT/EXPORT. The motivating use-case is the task of converting a Derby based MetaStore to MySQL. > Extend IMPORT/EXPORT to Support Metata Only Operations > -- > > Key: HIVE-2816 > URL: https://issues.apache.org/jira/browse/HIVE-2816 > Project: Hive > Issue Type: New Feature > Components: Import/Export >Reporter: Carl Steinbach > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2773) HiveStorageHandler.configureTableJobProperites() should let the handler know wether it is configuration for input or output
[ https://issues.apache.org/jira/browse/HIVE-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214962#comment-13214962 ] Francis Liu commented on HIVE-2773: --- If I add new methods, classes implementing the interface will have to implement the new methods anyway, why not just take the extra step of having them do a very little bit of refactoring (see my changes to HBaseStorageHandler)? Of course if most users are extending DefaultStorageHandler then doing as you say would be a smoother way to go about it. > HiveStorageHandler.configureTableJobProperites() should let the handler know > wether it is configuration for input or output > --- > > Key: HIVE-2773 > URL: https://issues.apache.org/jira/browse/HIVE-2773 > Project: Hive > Issue Type: Improvement >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hcatalog, storage_handler > Attachments: HIVE-2773.D1815.1.patch, HIVE-2773.patch > > > HiveStorageHandler.configureTableJobProperties() is called to allow the > storage handler to setup any properties that the underlying > inputformat/outputformat/serde may need. But the handler implementation does > not know whether it is being called for configuring input or output. This > makes it a problem for handlers which sets an external state. In the case of > HCatalog's HBase storageHandler, whenever a write needs to be configured we > create a write transaction which needs to be committed or aborted later on. > In this case configuring for both input and output each time > configureTableJobProperties() is called would not be desirable. This has > become an issue since HCatalog is dropping storageDrivers for SerDe and > StorageHandler (see HCATALOG-237). > My proposal is to replace configureTableJobProperties() with two methods: > configureInputJobProperties() > configureOutputJobProperties() > Each method will have the same signature. I cursory look at the code and I > believe changes should be straighforward also given that we are not really > changing anything just splitting responsibility. If the community is fine > with this approach I will go ahead and create a aptch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo reassigned HIVE-2811: - Assignee: Edward Capriolo > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2811: -- Attachment: HIVE-2811.D1905.1.patch edwardcapriolo requested code review of "HIVE-2811 [jira] Export LANG=en_US.UTF-8 to environment while running tests". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2811 UTF8 here Most OS has this value by default, but in some cases (on apache build machines) it has a different value making inputddl5.q test to fail. This variable should be exported while running the tests. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1905 AFFECTED FILES build-common.xml MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4053/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.D1905.1.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2773) HiveStorageHandler.configureTableJobProperites() should let the handler know wether it is configuration for input or output
[ https://issues.apache.org/jira/browse/HIVE-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214979#comment-13214979 ] Ashutosh Chauhan commented on HIVE-2773: bq. If I add new methods, classes implementing the interface will have to implement the new methods anyway, Adding or removing methods have different impacts from point of view of backward compatibility. If you add new methods in an interface then implementations using the older interface will continue to work even after they upgrade to new version of hive. That has no cost on people who have implemented the old interface. Only if they recompile it, then they have to implement new methods. On the other hand, if you remove methods, then their old implementation won't work anymore after upgrade. They have to recompile and this implies work on their part. This is the difference which gets factored in backward compatibility. So, as long as you are adding new methods in interface, I consider it to be backward compatible. But, if you are removing methods, then its not. Makes sense? > HiveStorageHandler.configureTableJobProperites() should let the handler know > wether it is configuration for input or output > --- > > Key: HIVE-2773 > URL: https://issues.apache.org/jira/browse/HIVE-2773 > Project: Hive > Issue Type: Improvement >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hcatalog, storage_handler > Attachments: HIVE-2773.D1815.1.patch, HIVE-2773.patch > > > HiveStorageHandler.configureTableJobProperties() is called to allow the > storage handler to setup any properties that the underlying > inputformat/outputformat/serde may need. But the handler implementation does > not know whether it is being called for configuring input or output. This > makes it a problem for handlers which sets an external state. In the case of > HCatalog's HBase storageHandler, whenever a write needs to be configured we > create a write transaction which needs to be committed or aborted later on. > In this case configuring for both input and output each time > configureTableJobProperties() is called would not be desirable. This has > become an issue since HCatalog is dropping storageDrivers for SerDe and > StorageHandler (see HCATALOG-237). > My proposal is to replace configureTableJobProperties() with two methods: > configureInputJobProperties() > configureOutputJobProperties() > Each method will have the same signature. I cursory look at the code and I > believe changes should be straighforward also given that we are not really > changing anything just splitting responsibility. If the community is fine > with this approach I will go ahead and create a aptch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-2811: -- Status: Patch Available (was: Open) > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.D1905.1.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-895) Add SerDe for Avro serialized data
[ https://issues.apache.org/jira/browse/HIVE-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214985#comment-13214985 ] Edward Capriolo commented on HIVE-895: -- This has a Apache V2 licence if anyone wants we should be able to just git --clone this and patch it in if we keep the licence file. > Add SerDe for Avro serialized data > -- > > Key: HIVE-895 > URL: https://issues.apache.org/jira/browse/HIVE-895 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Reporter: Jeff Hammerbacher >Assignee: Jakob Homan > > As Avro continues to mature, having a SerDe to allow HiveQL queries over Avro > data seems like a solid win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-895) Add SerDe for Avro serialized data
[ https://issues.apache.org/jira/browse/HIVE-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214993#comment-13214993 ] Jakob Homan commented on HIVE-895: -- yes, I've set aside some time early next week to get it into patch form. > Add SerDe for Avro serialized data > -- > > Key: HIVE-895 > URL: https://issues.apache.org/jira/browse/HIVE-895 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Reporter: Jeff Hammerbacher >Assignee: Jakob Homan > > As Avro continues to mature, having a SerDe to allow HiveQL queries over Avro > data seems like a solid win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2372) java.io.IOException: error=7, Argument list too long
[
https://issues.apache.org/jira/browse/HIVE-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214997#comment-13214997
]
Edward Capriolo commented on HIVE-2372:
---
Lets get this committed. I think we should introduce the variables. In the
normal case we want someone to get all the input they would expect. If they run
into an exception they can turn the variable on to get out of the problem.
Lets do this. Rebase your patch and add
hive.scriptoperator.truncate.env=true|false to control this feature. Default it
to false which would be how hive works now. I will review and commit.
> java.io.IOException: error=7, Argument list too long
>
>
> Key: HIVE-2372
> URL: https://issues.apache.org/jira/browse/HIVE-2372
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Sergey Tryuber
>Priority: Critical
> Attachments: HIVE-2372.1.patch.txt
>
>
> I execute a huge query on a table with a lot of 2-level partitions. There is
> a perl reducer in my query. Maps worked ok, but every reducer fails with the
> following exception:
> 2011-08-11 04:58:29,865 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> Executing [/usr/bin/perl, , ]
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> tablename=null
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> partname=null
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> alias=null
> 2011-08-11 04:58:29,935 FATAL ExecReducer:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":129390185139228,"reducesinkkey1":"8AF163CA6F"},"value":{"_col0":"8AF163CA6F","_col1":"2011-07-27
>
> 22:48:52","_col2":129390185139228,"_col3":2006,"_col4":4100,"_col5":"10017388=6","_col6":1063,"_col7":"NULL","_col8":"address.com","_col9":"NULL","_col10":"NULL"},"alias":0}
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Cannot
> initialize ScriptOperator
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:320)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
> at
> org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
> ... 7 more
> Caused by: java.io.IOException: Cannot run program "/usr/bin/perl":
> java.io.IOException: error=7, Argument list too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:279)
> ... 15 more
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument list
> too long
> at java.lang.UNIXProcess.(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 16 more
> It seems to me, I found the cause. ScriptOperator.java puts a lot of configs
> as environment variables to the child reduce process. One of variables is
> mapred.input.dir, which in my case more than 150KB. There are a huge amount
> of input directories in this variable. In short, the problem is that Linux
> (up to 2.6.23 kernel version) limits summary size of environment variables
> for child processes to 132KB. This problem could be solved by upgrading the
> kernel. But strings limitations still be 132KB per string in environment
> variable. So such huge variable doesn't work even on my home computer
> (2.6.32). You can read more information on
> (http://www.kernel.org/doc/man-pages/online/pages/man2/execve.2.htm
[jira] [Commented] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215004#comment-13215004 ] Phabricator commented on HIVE-2811: --- cwsteinbach has accepted the revision "HIVE-2811 [jira] Export LANG=en_US.UTF-8 to environment while running tests". Looks good. Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D1905 BRANCH svn > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.D1905.1.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2761) Remove lib/javaewah-0.3.jar
[ https://issues.apache.org/jira/browse/HIVE-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo reassigned HIVE-2761: - Assignee: Edward Capriolo > Remove lib/javaewah-0.3.jar > --- > > Key: HIVE-2761 > URL: https://issues.apache.org/jira/browse/HIVE-2761 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0, 0.8.1, 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > > After HIVE-2391 it is retrieved from maven repo via ivy, we can get rid of it > from our lib/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1271 - Fixed
Changes for Build #1270 [hashutosh] HIVE-2716 : Move retry logic in HiveMetaStore to a separe class (enis via hashutosh) Changes for Build #1271 All tests passed The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1271) Status: Fixed Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1271/ to view the results.
[jira] [Commented] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215020#comment-13215020 ] Carl Steinbach commented on HIVE-2811: -- @Ed: Can you please attach a copy of the patch and grant license rights to the ASF? Thanks! > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.D1905.1.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215027#comment-13215027 ] Edward Capriolo commented on HIVE-2811: --- Done. Are we going through all those fabricator steps just to have to put up the patch again by hand? Is there some fabricator setting I can check for this? > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.D1905.1.patch, HIVE-2811.D1905.2.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-2811: -- Attachment: HIVE-2811.D1905.2.patch > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.D1905.1.patch, HIVE-2811.D1905.2.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2761) Remove lib/javaewah-0.3.jar
[ https://issues.apache.org/jira/browse/HIVE-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2761: -- Attachment: HIVE-2761.D1911.1.patch edwardcapriolo requested code review of "HIVE-2761 [jira] Remove lib/javaewah-0.3.jar". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2761 Jar in SVN bad. Jar from maven good. After HIVE-2391 it is retrieved from maven repo via ivy, we can get rid of it from our lib/ TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1911 AFFECTED FILES lib/javaewah-0.3.jar lib/javaewah.LICENSE ivy/libraries.properties ql/ivy.xml ql/build.xml MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4059/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Remove lib/javaewah-0.3.jar > --- > > Key: HIVE-2761 > URL: https://issues.apache.org/jira/browse/HIVE-2761 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0, 0.8.1, 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2761.D1911.1.patch > > > After HIVE-2391 it is retrieved from maven repo via ivy, we can get rid of it > from our lib/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215053#comment-13215053 ] Carl Steinbach commented on HIVE-2811: -- Unfortunately, yes. I think John is looking into making arc do this automatically. In the meantime, sorry for the inconvenience. > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.D1905.1.patch, HIVE-2811.D1905.2.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215065#comment-13215065 ] Phabricator commented on HIVE-2811: --- ashutoshc has commented on the revision "HIVE-2811 [jira] Export LANG=en_US.UTF-8 to environment while running tests". INLINE COMMENTS build-common.xml:67 Is it en_US.UTF8 or en_US.UTF-8? I think latter. REVISION DETAIL https://reviews.facebook.net/D1905 BRANCH svn > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.D1905.1.patch, HIVE-2811.D1905.2.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2768) Add a getAuthorizationProvider to HiveStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-2768. Resolution: Fixed Fix Version/s: 0.9.0 Committed to trunk. Thanks, Francis! > Add a getAuthorizationProvider to HiveStorageHandler > > > Key: HIVE-2768 > URL: https://issues.apache.org/jira/browse/HIVE-2768 > Project: Hive > Issue Type: Task > Components: HBase Handler >Reporter: Alan Gates >Assignee: Francis Liu > Fix For: 0.9.0 > > Attachments: HIVE-2768.D1869.1.patch > > > In version 0.92 HBase supports ACLs for tables. In HCatalog, since we > delegate security to the underlying storage layer, we would like to be able > to obtain a HiveAuthorizationProvider specific to a HiveStorageHandler > instance. This can be done by adding a getAuthorizationProvider method to > HiveStorageHandler. In the case where Hive is configured to use the > DefaultHiveAuthorizationProvider this call will return the same default > provider, since Hive handles all of the authorization itself in that case. > In the case where it is configured to use the HCatAuthorizationProvider, it > would return an instance specific to the underlying storage. > For more background on this proposed change see HCATALOG-237 and > https://cwiki.apache.org/confluence/display/HCATALOG/Hcat+Security+Design -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2768) Add a getAuthorizationProvider to HiveStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HIVE-2768: -- Attachment: HIVE-2768.D1869.1.patch > Add a getAuthorizationProvider to HiveStorageHandler > > > Key: HIVE-2768 > URL: https://issues.apache.org/jira/browse/HIVE-2768 > Project: Hive > Issue Type: Task > Components: HBase Handler >Reporter: Alan Gates >Assignee: Francis Liu > Fix For: 0.9.0 > > Attachments: HIVE-2768.D1869.1.patch, HIVE-2768.D1869.1.patch > > > In version 0.92 HBase supports ACLs for tables. In HCatalog, since we > delegate security to the underlying storage layer, we would like to be able > to obtain a HiveAuthorizationProvider specific to a HiveStorageHandler > instance. This can be done by adding a getAuthorizationProvider method to > HiveStorageHandler. In the case where Hive is configured to use the > DefaultHiveAuthorizationProvider this call will return the same default > provider, since Hive handles all of the authorization itself in that case. > In the case where it is configured to use the HCatAuthorizationProvider, it > would return an instance specific to the underlying storage. > For more background on this proposed change see HCATALOG-237 and > https://cwiki.apache.org/confluence/display/HCATALOG/Hcat+Security+Design -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215093#comment-13215093 ] Phabricator commented on HIVE-2811: --- cwsteinbach has requested changes to the revision "HIVE-2811 [jira] Export LANG=en_US.UTF-8 to environment while running tests". Good catch. Ed, can you please make this change? Thanks. REVISION DETAIL https://reviews.facebook.net/D1905 BRANCH svn > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.D1905.1.patch, HIVE-2811.D1905.2.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2817) Drop any table even without privilege
Drop any table even without privilege
-
Key: HIVE-2817
URL: https://issues.apache.org/jira/browse/HIVE-2817
Project: Hive
Issue Type: Bug
Affects Versions: 0.7.1
Reporter: Benyi Wang
You can drop any table if you use fully qualified name 'database.table' even
you don't have any previlige.
{code}
hive> set hive.security.authorization.enabled=true;
hive> revoke all on default from user test_user;
hive> drop table abc;
hive> drop table abc;
Authorization failed:No privilege 'Drop' found for outputs { database:default,
table:abc}. Use show grant to get more details.
hive> drop table default.abc;
OK
Time taken: 0.13 seconds
{code}
The table and the file in {{/usr/hive/warehouse}} or external file will be
deleted. If you don't have hadoop access permission on {{/usr/hive/warehouse}}
or external files, you will see a hadoop access error
{code}
12/02/23 15:35:35 ERROR hive.log:
org.apache.hadoop.security.AccessControlException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=test_user, access=WRITE, inode="/user/myetl":myetl:etl:drwxr-xr-x
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2818) Create table checks the current database privilege
Create table checks the current database privilege
--
Key: HIVE-2818
URL: https://issues.apache.org/jira/browse/HIVE-2818
Project: Hive
Issue Type: Bug
Affects Versions: 0.7.1
Reporter: Benyi Wang
Hive seems check the current database to determine the privilege of a statement
when you use fully qualified name like 'database.table'
{code}
hive> set hive.security.authorization.enabled=true;
hive> create database test_db;
hive> grant all on database test_db to user test_user;
hive> revoke all on database default from test_user;
hive> use default;
hive> create table test_db.new_table (id int);
Authorization failed:No privilege 'Create' found for outputs {
database:default}. Use show grant to get more details.
hive> use test_db;
hive> create table test_db.new_table (id int);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-2811: -- Attachment: HIVE-2811.3.patch Third try is a charm > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.3.patch, HIVE-2811.D1905.1.patch, > HIVE-2811.D1905.2.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2811) Export LANG=en_US.UTF-8 to environment while running tests
[ https://issues.apache.org/jira/browse/HIVE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-2811: -- Attachment: HIVE-2811.3.patch > Export LANG=en_US.UTF-8 to environment while running tests > -- > > Key: HIVE-2811 > URL: https://issues.apache.org/jira/browse/HIVE-2811 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2811.3.patch, HIVE-2811.3.patch, > HIVE-2811.D1905.1.patch, HIVE-2811.D1905.2.patch > > > Most OS has this value by default, but in some cases (on apache build > machines) it has a different value making inputddl5.q test to fail. This > variable should be exported while running the tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2819) Closed range scans on hbase keys
Closed range scans on hbase keys - Key: HIVE-2819 URL: https://issues.apache.org/jira/browse/HIVE-2819 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan This patch pushes range scans on keys of closed form into hbase -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2819) Closed range scans on hbase keys
[ https://issues.apache.org/jira/browse/HIVE-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2819: -- Attachment: HIVE-2819.D1923.1.patch ashutoshc requested code review of "HIVE-2819 [jira] Closed range scans on hbase keys". Reviewers: JIRA Last bits for range scans on keys. This patch pushes range scans on keys of closed form into hbase TEST PLAN Added new tests in ppd_key_ranges.q REVISION DETAIL https://reviews.facebook.net/D1923 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/test/queries/ppd_key_ranges.q hbase-handler/src/test/results/ppd_key_ranges.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4077/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Closed range scans on hbase keys > - > > Key: HIVE-2819 > URL: https://issues.apache.org/jira/browse/HIVE-2819 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-2819.D1923.1.patch > > > This patch pushes range scans on keys of closed form into hbase -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2818) Create table checks the current database privilege
[
https://issues.apache.org/jira/browse/HIVE-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-2818:
-
Component/s: Authorization
> Create table checks the current database privilege
> --
>
> Key: HIVE-2818
> URL: https://issues.apache.org/jira/browse/HIVE-2818
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.7.1
>Reporter: Benyi Wang
>
> Hive seems check the current database to determine the privilege of a
> statement when you use fully qualified name like 'database.table'
>
> {code}
> hive> set hive.security.authorization.enabled=true;
> hive> create database test_db;
> hive> grant all on database test_db to user test_user;
> hive> revoke all on database default from test_user;
> hive> use default;
> hive> create table test_db.new_table (id int);
> Authorization failed:No privilege 'Create' found for outputs {
> database:default}. Use show grant to get more details.
> hive> use test_db;
> hive> create table test_db.new_table (id int);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2761) Remove lib/javaewah-0.3.jar
[ https://issues.apache.org/jira/browse/HIVE-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215261#comment-13215261 ] Phabricator commented on HIVE-2761: --- ashutoshc has accepted the revision "HIVE-2761 [jira] Remove lib/javaewah-0.3.jar". +1 Running tests. REVISION DETAIL https://reviews.facebook.net/D1911 BRANCH svn > Remove lib/javaewah-0.3.jar > --- > > Key: HIVE-2761 > URL: https://issues.apache.org/jira/browse/HIVE-2761 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0, 0.8.1, 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2761.D1911.1.patch > > > After HIVE-2391 it is retrieved from maven repo via ivy, we can get rid of it > from our lib/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2761) Remove lib/javaewah-0.3.jar
[ https://issues.apache.org/jira/browse/HIVE-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215265#comment-13215265 ] Ashutosh Chauhan commented on HIVE-2761: @Ed, Can you upload the patch on jira granting perms? > Remove lib/javaewah-0.3.jar > --- > > Key: HIVE-2761 > URL: https://issues.apache.org/jira/browse/HIVE-2761 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0, 0.8.1, 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2761.D1911.1.patch > > > After HIVE-2391 it is retrieved from maven repo via ivy, we can get rid of it > from our lib/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2820) Invalid tag is used for MapJoinProcessor
Invalid tag is used for MapJoinProcessor
Key: HIVE-2820
URL: https://issues.apache.org/jira/browse/HIVE-2820
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.9.0
Environment: ubuntu
Reporter: Navis
Assignee: Navis
Fix For: 0.9.0
Testing HIVE-2810, I've found tag and alias are used in very confusing manner.
For example, query below fails..
{code}
hive> set hive.auto.convert.join=true;
hive> select /*+ STREAMTABLE(a) */ * from myinput1 a join myinput1 b on
a.key=b.key join myinput1 c on a.key=c.key;
Total MapReduce jobs = 4
Ended Job = 1667415037, job is filtered out (removed at runtime).
Ended Job = 1739566906, job is filtered out (removed at runtime).
Ended Job = 1113337780, job is filtered out (removed at runtime).
12/02/24 10:27:14 WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml
found on the CLASSPATH at /home/navis/hive/conf/hive-default.xml
Execution log at:
/tmp/navis/navis_20120224102727_cafe0d8d-9b21-441d-bd4e-b83303b31cdc.log
2012-02-24 10:27:14 Starting to launch local task to process map join;
maximum memory = 932118528
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.processOp(HashTableSinkOperator.java:312)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at
org.apache.hadoop.hive.ql.exec.MapredLocalTask.startForward(MapredLocalTask.java:325)
at
org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:272)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:685)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Execution failed with exit status: 2
Obtaining error information
{code}
Failed task has a plan which doesn't make sense.
{noformat}
Stage: Stage-8
Map Reduce Local Work
Alias -> Map Local Tables:
b
Fetch Operator
limit: -1
c
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
b
TableScan
alias: b
HashTable Sink Operator
condition expressions:
0 {key} {value}
1 {key} {value}
2 {key} {value}
handleSkewJoin: false
keys:
0 [Column[key]]
1 [Column[key]]
2 [Column[key]]
Position of Big Table: 0
c
TableScan
alias: c
Map Join Operator
condition map:
Inner Join 0 to 1
Inner Join 0 to 2
condition expressions:
0 {key} {value}
1 {key} {value}
2 {key} {value}
handleSkewJoin: false
keys:
0 [Column[key]]
1 [Column[key]]
2 [Column[key]]
outputColumnNames: _col0, _col1, _col4, _col5, _col8, _col9
Position of Big Table: 0
Select Operator
expressions:
expr: _col0
type: int
expr: _col1
type: int
expr: _col4
type: int
expr: _col5
type: int
expr: _col8
type: int
expr: _col9
type: int
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Stage: Stage-4
Map Reduce
Alias -> Map Operator Tree:
a
TableScan
alias: a
HashTable Sink Operator
condition expressions:
0 {key} {value}
1 {key} {value}
2 {key} {value}
handleSkew
[jira] [Commented] (HIVE-2768) Add a getAuthorizationProvider to HiveStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215326#comment-13215326 ] Hudson commented on HIVE-2768: -- Integrated in Hive-trunk-h0.21 #1272 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1272/]) HIVE-2768: Add a getAuthorizationProvider to HiveStorageHandler (toffer via hashutosh) (Revision 1292969) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1292969 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DefaultStorageHandler.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java > Add a getAuthorizationProvider to HiveStorageHandler > > > Key: HIVE-2768 > URL: https://issues.apache.org/jira/browse/HIVE-2768 > Project: Hive > Issue Type: Task > Components: HBase Handler >Reporter: Alan Gates >Assignee: Francis Liu > Fix For: 0.9.0 > > Attachments: HIVE-2768.D1869.1.patch, HIVE-2768.D1869.1.patch > > > In version 0.92 HBase supports ACLs for tables. In HCatalog, since we > delegate security to the underlying storage layer, we would like to be able > to obtain a HiveAuthorizationProvider specific to a HiveStorageHandler > instance. This can be done by adding a getAuthorizationProvider method to > HiveStorageHandler. In the case where Hive is configured to use the > DefaultHiveAuthorizationProvider this call will return the same default > provider, since Hive handles all of the authorization itself in that case. > In the case where it is configured to use the HCatAuthorizationProvider, it > would return an instance specific to the underlying storage. > For more background on this proposed change see HCATALOG-237 and > https://cwiki.apache.org/confluence/display/HCATALOG/Hcat+Security+Design -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2821) union with two mapjoin will throw NPE
union with two mapjoin will throw NPE --- Key: HIVE-2821 URL: https://issues.apache.org/jira/browse/HIVE-2821 Project: Hive Issue Type: Bug Affects Versions: 0.7.0 Environment: Linux zongren-VirtualBox 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux java version "1.6.0_25" hadoop-0.20.2-cdh3u0 hive-0.7.0-cdh3u0 Reporter: caofangkun Priority: Critical create table src (key string, value string); create table src1 (key string, value string); select count(*) from ( select /+mapjoin(b)/ a.* from src a join src1 b on a.key=b.key where a.key=48 union all select /+mapjoin(bb)/ aa.* from src aa join src1 bb on aa.key=bb.key where aa.key=100 ) t; FAILED: Hive Internal Error: java.lang.NullPointerException(null) java.lang.NullPointerException at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:156) at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.setTaskPlan(GenMapRedUtils.java:553) at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.setTaskPlan(GenMapRedUtils.java:514) at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.initPlan(GenMapRedUtils.java:125) at org.apache.hadoop.hive.ql.optimizer.GenMRRedSink1.process(GenMRRedSink1.java:76) at org.apache.hadoop.hive.ql.optimizer.GenMRRedSink3.process(GenMRRedSink3.java:64) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:55) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:6946) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:7247) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:240) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:904) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:279) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:228) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:417) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:350) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:451) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:461) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:585) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2761) Remove lib/javaewah-0.3.jar
[ https://issues.apache.org/jira/browse/HIVE-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2761: -- Attachment: HIVE-2761.D1911.2.patch edwardcapriolo updated the revision "HIVE-2761 [jira] Remove lib/javaewah-0.3.jar". Reviewers: JIRA, ashutoshc Found one more instance of javaewah hardcoded. REVISION DETAIL https://reviews.facebook.net/D1911 AFFECTED FILES lib/javaewah-0.3.jar lib/javaewah.LICENSE ivy/libraries.properties build-common.xml ql/ivy.xml ql/build.xml > Remove lib/javaewah-0.3.jar > --- > > Key: HIVE-2761 > URL: https://issues.apache.org/jira/browse/HIVE-2761 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0, 0.8.1, 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2761.D1911.1.patch, HIVE-2761.D1911.2.patch > > > After HIVE-2391 it is retrieved from maven repo via ivy, we can get rid of it > from our lib/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2761) Remove lib/javaewah-0.3.jar
[ https://issues.apache.org/jira/browse/HIVE-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-2761: -- Attachment: HIVE-2761.2.patch.txt Found one more instance of javaEwah > Remove lib/javaewah-0.3.jar > --- > > Key: HIVE-2761 > URL: https://issues.apache.org/jira/browse/HIVE-2761 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0, 0.8.1, 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2761.2.patch.txt, HIVE-2761.D1911.1.patch, > HIVE-2761.D1911.2.patch > > > After HIVE-2391 it is retrieved from maven repo via ivy, we can get rid of it > from our lib/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2761) Remove lib/javaewah-0.3.jar
[ https://issues.apache.org/jira/browse/HIVE-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215418#comment-13215418 ] Ashutosh Chauhan commented on HIVE-2761: There is one more in .classpath > Remove lib/javaewah-0.3.jar > --- > > Key: HIVE-2761 > URL: https://issues.apache.org/jira/browse/HIVE-2761 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0, 0.8.1, 0.9.0 >Reporter: Ashutosh Chauhan >Assignee: Edward Capriolo > Fix For: 0.9.0 > > Attachments: HIVE-2761.2.patch.txt, HIVE-2761.D1911.1.patch, > HIVE-2761.D1911.2.patch > > > After HIVE-2391 it is retrieved from maven repo via ivy, we can get rid of it > from our lib/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2820) Invalid tag is used for MapJoinProcessor
[
https://issues.apache.org/jira/browse/HIVE-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-2820:
--
Attachment: HIVE-2820.D1935.1.patch
navis requested code review of "HIVE-2820 [jira] Invalid tag is used for
MapJoinProcessor".
Reviewers: JIRA
DPAL-889 Invalid tag is used for MapJoinProcessor
Testing HIVE-2810, I've found tag and alias are used in very confusing
manner. For example, query below fails..
hive> set hive.auto.convert.join=true;
hive> select /*+ STREAMTABLE(a) */ * from myinput1 a join myinput1 b on
a.key=b.key join myinput1 c on a.key=c.key;
Total MapReduce jobs = 4
Ended Job = 1667415037, job is filtered out (removed at runtime).
Ended Job = 1739566906, job is filtered out (removed at runtime).
Ended Job = 1113337780, job is filtered out (removed at runtime).
12/02/24 10:27:14 WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml
found on the CLASSPATH at /home/navis/hive/conf/hive-default.xml
Execution log at:
/tmp/navis/navis_20120224102727_cafe0d8d-9b21-441d-bd4e-b83303b31cdc.log
2012-02-24 10:27:14 Starting to launch local task to process map join;
maximum memory = 932118528
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.processOp(HashTableSinkOperator.java:312)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at
org.apache.hadoop.hive.ql.exec.MapredLocalTask.startForward(MapredLocalTask.java:325)
at
org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:272)
at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:685)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Execution failed with exit status: 2
Obtaining error information
Failed task has a plan which doesn't make sense.
Stage: Stage-8
Map Reduce Local Work
Alias -> Map Local Tables:
b
Fetch Operator
limit: -1
c
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
b
TableScan
alias: b
HashTable Sink Operator
condition expressions:
0 {key} {value}
1 {key} {value}
2 {key} {value}
handleSkewJoin: false
keys:
0 [Column[key]]
1 [Column[key]]
2 [Column[key]]
Position of Big Table: 0
c
TableScan
alias: c
Map Join Operator
condition map:
Inner Join 0 to 1
Inner Join 0 to 2
condition expressions:
0 {key} {value}
1 {key} {value}
2 {key} {value}
handleSkewJoin: false
keys:
0 [Column[key]]
1 [Column[key]]
2 [Column[key]]
outputColumnNames: _col0, _col1, _col4, _col5, _col8, _col9
Position of Big Table: 0
Select Operator
expressions:
expr: _col0
type: int
expr: _col1
type: int
expr: _col4
type: int
expr: _col5
type: int
expr: _col8
type: int
expr: _col9
type: int
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Stage: Stage-4
Map Reduce
Alias -> Map Operator Tree:
a
TableScan
alias: a
HashTable Sink Operator
condition expressions:
0 {key} {value}
1 {key} {value}
2 {key} {value
[jira] [Commented] (HIVE-2820) Invalid tag is used for MapJoinProcessor
[
https://issues.apache.org/jira/browse/HIVE-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215464#comment-13215464
]
Phabricator commented on HIVE-2820:
---
navis has commented on the revision "HIVE-2820 [jira] Invalid tag is used for
MapJoinProcessor".
Passed all the *join* tests in client-positive. Full test result would be
posted shortly.
REVISION DETAIL
https://reviews.facebook.net/D1935
> Invalid tag is used for MapJoinProcessor
>
>
> Key: HIVE-2820
> URL: https://issues.apache.org/jira/browse/HIVE-2820
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.9.0
> Environment: ubuntu
>Reporter: Navis
>Assignee: Navis
> Fix For: 0.9.0
>
> Attachments: HIVE-2820.D1935.1.patch
>
>
> Testing HIVE-2810, I've found tag and alias are used in very confusing
> manner. For example, query below fails..
> {code}
> hive> set hive.auto.convert.join=true;
>
> hive> select /*+ STREAMTABLE(a) */ * from myinput1 a join myinput1 b on
> a.key=b.key join myinput1 c on a.key=c.key;
> Total MapReduce jobs = 4
> Ended Job = 1667415037, job is filtered out (removed at runtime).
> Ended Job = 1739566906, job is filtered out (removed at runtime).
> Ended Job = 1113337780, job is filtered out (removed at runtime).
> 12/02/24 10:27:14 WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml
> found on the CLASSPATH at /home/navis/hive/conf/hive-default.xml
> Execution log at:
> /tmp/navis/navis_20120224102727_cafe0d8d-9b21-441d-bd4e-b83303b31cdc.log
> 2012-02-24 10:27:14 Starting to launch local task to process map join;
> maximum memory = 932118528
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.processOp(HashTableSinkOperator.java:312)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.startForward(MapredLocalTask.java:325)
> at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:272)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:685)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Execution failed with exit status: 2
> Obtaining error information
> {code}
> Failed task has a plan which doesn't make sense.
> {noformat}
> Stage: Stage-8
> Map Reduce Local Work
> Alias -> Map Local Tables:
> b
> Fetch Operator
> limit: -1
> c
> Fetch Operator
> limit: -1
> Alias -> Map Local Operator Tree:
> b
> TableScan
> alias: b
> HashTable Sink Operator
> condition expressions:
> 0 {key} {value}
> 1 {key} {value}
> 2 {key} {value}
> handleSkewJoin: false
> keys:
> 0 [Column[key]]
> 1 [Column[key]]
> 2 [Column[key]]
> Position of Big Table: 0
> c
> TableScan
> alias: c
> Map Join Operator
> condition map:
>Inner Join 0 to 1
>Inner Join 0 to 2
> condition expressions:
> 0 {key} {value}
> 1 {key} {value}
> 2 {key} {value}
> handleSkewJoin: false
> keys:
> 0 [Column[key]]
> 1 [Column[key]]
> 2 [Column[key]]
> outputColumnNames: _col0, _col1, _col4, _col5, _col8, _col9
> Position of Big Table: 0
> Select Operator
> expressions:
> expr: _col0
> type: int
> expr: _col1
> type: int
> expr: _col4
> type: int
> expr: _col5
> type: int
> expr: _col8
> type: int
>
