[jira] Resolved: (HIVE-932) enforce bucketing
[ https://issues.apache.org/jira/browse/HIVE-932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang resolved HIVE-932. --- Resolution: Fixed Already fixed in trunk by namit. > enforce bucketing > - > > Key: HIVE-932 > URL: https://issues.apache.org/jira/browse/HIVE-932 > Project: Hadoop Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Namit Jain >Assignee: He Yongqiang > > If a table is bucketed and the number of buckets are also specified, it > should be enforced. > If a query tries to insert into the table which violates the bucketing, it > should fail. > Also, the number of reducers should be the same as the number of buckets, and > be independent of the input size -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1258) set merge files to files when bucketing/sorting is being enforced
set merge files to files when bucketing/sorting is being enforced - Key: HIVE-1258 URL: https://issues.apache.org/jira/browse/HIVE-1258 Project: Hadoop Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: He Yongqiang -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1257) joins between HBase tables and other tables (whether HBase or not) are broken
[ https://issues.apache.org/jira/browse/HIVE-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847184#action_12847184 ] John Sichi commented on HIVE-1257: -- I also need to add new tests for the real patch submission. > joins between HBase tables and other tables (whether HBase or not) are broken > - > > Key: HIVE-1257 > URL: https://issues.apache.org/jira/browse/HIVE-1257 > Project: Hadoop Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.6.0 > > Attachments: HIVE-1257.prelim.patch > > > Details in > http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201003.mbox/%3c9a53dde1fe082f4d952fdf20ac87e21f021f3...@exchange2.t8design.com%3e -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1257) joins between HBase tables and other tables (whether HBase or not) are broken
[ https://issues.apache.org/jira/browse/HIVE-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1257: - Attachment: HIVE-1257.prelim.patch Preliminary patch which fixes the test queries; I haven't run through regression tests yet. > joins between HBase tables and other tables (whether HBase or not) are broken > - > > Key: HIVE-1257 > URL: https://issues.apache.org/jira/browse/HIVE-1257 > Project: Hadoop Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: John Sichi > Fix For: 0.6.0 > > Attachments: HIVE-1257.prelim.patch > > > Details in > http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201003.mbox/%3c9a53dde1fe082f4d952fdf20ac87e21f021f3...@exchange2.t8design.com%3e -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1257) joins between HBase tables and other tables (whether HBase or not) are broken
[
https://issues.apache.org/jira/browse/HIVE-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847155#action_12847155
]
John Sichi commented on HIVE-1257:
--
Copying details from mailing list thread:
>From "Daniel Pruckler"
Subject Problem when joining hbase tables in hive
DateThu, 18 Mar 2010 16:50:22 GMT
We're just starting with the HBaseIntegration, and are really excited about it,
great work
on this. Thanks
We are running into a problem when joining two hbase tables. It seems to
usually work when
joining one hbase table with one hive table, but not always. Let me know if
I'm doing something
wrong.
My workaround for right now is to create new tables in hive and overwrite that
table with
the external hbase table, and then join from the new tables. In our situation
this is working
as the one large table we've got seems to work just fine in hbase, we just need
to overwrite
the smaller join tables before doing a join. I have one join example at the
end of this email
that fails when joining an hbase and hive table.
Here's what I did:
Create the HBase tables:
create 'users', 'info'
put 'users', 'user1', 'info:state', 'IA'
put 'users', 'user1', 'info:country', 'USA'
put 'users', 'user1', 'info:country_id', '0'
create 'states', 'state'
put 'states', 'IA', 'state:name', 'Iowa'
create 'countries', 'info'
put 'countries', 'USA', 'info:name', 'United States'
put 'countries', 'USA', 'info:country', 'USA'
put 'countries', 'USA', 'info:country_id', '1'
Create the hive tables:
CREATE EXTERNAL TABLE users(key string, state string, country string,
country_id int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.key.type" = "string",
"hbase.columns.mapping" = "info:state,info:country,info:country_id",
"hbase.table.name" = "users"
);
CREATE EXTERNAL TABLE states(key string, name string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.key.type" = "string",
"hbase.columns.mapping" = "state:name",
"hbase.table.name" = "states"
);
CREATE EXTERNAL TABLE countries(key string, name string, country string,
country_id int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.key.type" = "string",
"hbase.columns.mapping" = "info:name,info:country,info:country_id",
"hbase.table.name" = "countries"
);
Then try to join, this returns no results:
SELECT u.key, u.country, c.name, c.key FROM users u JOIN countries c ON
(u.country = c.key);
OK
Time taken: 26.75 seconds
This returns results, but not what's expected, it seems like it joins the user
table with
itself
SELECT u.key, u.country, c.name, c.key FROM users u JOIN countries c ON
(u.country = c.country);
OK
user1 USANULL user1
Time taken: 24.867 seconds
This also returns results (it should not as the country_ids do not match)
SELECT u.key, u.country, c.name, c.key FROM users u JOIN countries c ON
(u.country_id = c.country_id);
OK
user1 USANULL user1
Time taken: 22.868 seconds
I would guess this is part of the same issue, trying to join anything in state
with user results
in an error (state does not have an 'info' column family)
hive> SELECT u.key, u.state, s.name FROM users u JOIN states s ON (u.state =
s.key);
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Starting Job = job_201003150937_0096, Tracking URL =
http://localhost:50030/jobdetails.jsp?jobid=job_201003150937_0096
Kill Command = /usr/lib/hadoop/bin/../bin/hadoop job
-Dmapred.job.tracker=localhost:9001
-kill job_201003150937_0096
2010-03-18 10:43:28,730 Stage-1 map = 0%, reduce = 0%
2010-03-18 10:43:34,908 Stage-1 map = 50%, reduce = 0%
2010-03-18 10:43:44,020 Stage-1 map = 50%, reduce = 17%
2010-03-18 10:44:02,146 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201003150937_0096 with errors
Task with the most failures(4):
-
Task ID:
task_201003150937_0096_m_01
URL:
http://localhost:50030/taskdetails.jsp?jobid=job_201003150937_0096&tipid=task_201003150937_0096_m_01
-
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.ExecDriver
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException:
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException:
Column family state does not exist in region users,,1268926579188 in table
{NAME => 'users',
FAMILIES => [{NAME => 'info', COMPRESSION => 'NONE',
[jira] Created: (HIVE-1257) joins between HBase tables and other tables (whether HBase or not) are broken
joins between HBase tables and other tables (whether HBase or not) are broken - Key: HIVE-1257 URL: https://issues.apache.org/jira/browse/HIVE-1257 Project: Hadoop Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.6.0 Details in http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201003.mbox/%3c9a53dde1fe082f4d952fdf20ac87e21f021f3...@exchange2.t8design.com%3e -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1256) fix Hive logo img tag to avoid stretching
fix Hive logo img tag to avoid stretching - Key: HIVE-1256 URL: https://issues.apache.org/jira/browse/HIVE-1256 Project: Hadoop Hive Issue Type: Bug Components: Documentation Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Zheng Shao Fix For: 0.6.0 >From comment on HIVE-422: Aaron Newton added a comment - 17/Mar/10 02:32 AM Hey guys, I saw this article on TC today: http://techcrunch.com/2010/03/16/big-data-freedom/ and noticed the hive logo was all out of whack - all stretched out. Then I noticed it's like that on the Hive home page. Can someone fix the dimensions of the image tag? It looks kinda bad (and people are apparently using it like that elsewhere as seen on the TC article). http://hadoop.apache.org/hive/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1256) fix Hive logo img tag to avoid stretching
[ https://issues.apache.org/jira/browse/HIVE-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847095#action_12847095 ] John Sichi commented on HIVE-1256: -- Zheng, do you know where the Hive website sources live, and how we can update those? > fix Hive logo img tag to avoid stretching > - > > Key: HIVE-1256 > URL: https://issues.apache.org/jira/browse/HIVE-1256 > Project: Hadoop Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.6.0 >Reporter: John Sichi >Assignee: Zheng Shao > Fix For: 0.6.0 > > > From comment on HIVE-422: > Aaron Newton added a comment - 17/Mar/10 02:32 AM > Hey guys, > I saw this article on TC today: > http://techcrunch.com/2010/03/16/big-data-freedom/ > and noticed the hive logo was all out of whack - all stretched out. Then I > noticed it's like that on the Hive home page. Can someone fix the dimensions > of the image tag? It looks kinda bad (and people are apparently using it like > that elsewhere as seen on the TC article). > http://hadoop.apache.org/hive/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-422) Logo for Hive
[ https://issues.apache.org/jira/browse/HIVE-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847096#action_12847096 ] John Sichi commented on HIVE-422: - Opened HIVE-1256 to get attention for this. > Logo for Hive > - > > Key: HIVE-422 > URL: https://issues.apache.org/jira/browse/HIVE-422 > Project: Hadoop Hive > Issue Type: Improvement >Reporter: Aaron Newton >Assignee: Aaron Newton >Priority: Trivial > Fix For: 0.3.1, 0.4.0 > > Attachments: hive & job tracker icons (font outlines).pdf, hive.png > > > Greetings fine Hadoop peoples, > While working on a few projects here at Cloudera we found ourselves wanting > for some sort of icon for both the JobTracker and for Hive. After checking on > the project page for Hive (the JobTracker doesn't really have one) and > finding that these items have no icons, we rolled up our sleeves and made > some. We'd like to contribute these to the project, so if you want 'em, > they're all yours. > I opened a ticket here for both Hive and JobTracker logos: > https://issues.apache.org/jira/browse/HADOOP-5683 > But Ashish Thusoo suggested I open a ticket here for the hive logo > specifically. See attached. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-365) Create Table to support multiple levels of delimiters
[
https://issues.apache.org/jira/browse/HIVE-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847048#action_12847048
]
Zheng Shao commented on HIVE-365:
-
I am thinking something like:
{code}
CREATE TABLE nested(array_of_arrays ARRAY < ARRAY >, map_of_maps MAP <
STRING, MAP < INT, INT> > )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001' '\002' '\003' '\004' '\005';
{code}
Basically allowing multiple separators after "FIELDS TERMINATED".
The top level (fields) consumes 1 level of separators. Each level of array
consumes 1 level of separators, while each level of map consumes 2.
> Create Table to support multiple levels of delimiters
> -
>
> Key: HIVE-365
> URL: https://issues.apache.org/jira/browse/HIVE-365
> Project: Hadoop Hive
> Issue Type: New Feature
>Reporter: Zheng Shao
>
> From HIVE-337, the SerDe layer now supports multiple-levels of delimiters,
> for the purpose of supporting nested map/array/struct.
> Array(the same as List) and struct consume a single level of separator, and
> Map consumes 2 levels.
> DDL (Create Table) needs to allow users to specify multiple levels of
> delimiters in order to take the advantage of this new feature.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1219) More robust handling of metastore connection failures
[
https://issues.apache.org/jira/browse/HIVE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847051#action_12847051
]
Prasad Chakka commented on HIVE-1219:
-
@paul, makes sense to add separator in the constants. otherwise people are
bound to make mistakes.
ObjectStore.java
# good idea to add a lock. But only the first thread that encountered problems
with its PersistentManager object should try to recreate
PersistenetManagerFactory and not the subsequent threads. May be you can create
a new PMF only when PM's reference to PMF match and otherwise. Though I doubt
how much of this will be real problem for metastore server.
HiveMetastore.java
# updateConnectionURL() throw back the exception instead of just logging and
returning false. Also there is no need to log when rethrowing the exception.
Caller would log it anyways if needed (in initConnectionUrlHook)
Do you reload hive conf on every retry?
Otherwise patch looks good.
> More robust handling of metastore connection failures
> -
>
> Key: HIVE-1219
> URL: https://issues.apache.org/jira/browse/HIVE-1219
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Metastore
>Reporter: Paul Yang
>Assignee: Paul Yang
> Fix For: 0.6.0
>
> Attachments: HIVE-1219.1.patch, HIVE-1219.2.patch, HIVE-1219.3.patch,
> HIVE-1219.4.patch, HIVE-1219.5.patch
>
>
> Currently, if metastore's connection to the datastore is broken, the query
> fails and the exception such as the following is thrown
> {code}
> 2010-01-28 11:50:20,885 ERROR exec.MoveTask
> (SessionState.java:printError(248)) - Failed with exception Unable to fetch
> table tmp_table
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table
> tmp_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:362)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:333)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:112)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:99)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:64)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:582)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:462)
> at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:324)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:200)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:256)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: javax.jdo.JDODataStoreException: Communications link failure
> Last packet sent to the server was 1 ms ago.
> NestedThrowables:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
> failure
> Last packet sent to the server was 1 ms ago.
> at
> org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:289)
> {code}
> In order to reduce the impact of transient network issues and momentarily
> unavailable datastores, two possible improvements are:
> 1. Retrying the metastore command in case of connection failure before
> propagating up the exception.
> 2. Retrieving the datastore hostname / connection URL through the use of an
> extension. This extension would be useful in the case where a remote service
> maintained the location of the currently available datastore. In case of
> hostname changes or failovers to a backup datastore, the extension would
> allow hive clients to run without manual intervention.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1255) Add mathamatical UDFs PI, E, degrees, radians,tan, and atan
Add mathamatical UDFs PI, E, degrees, radians,tan, and atan --- Key: HIVE-1255 URL: https://issues.apache.org/jira/browse/HIVE-1255 Project: Hadoop Hive Issue Type: New Feature Components: Query Processor Affects Versions: 0.3.1, 0.4.2, 0.5.1, 0.6.0 Reporter: Edward Capriolo Assignee: Edward Capriolo Fix For: 0.6.0 Add support for PI, E, degrees, radians,tan, and atan -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1252) Task breaking bug when breaking after a filter operator
[ https://issues.apache.org/jira/browse/HIVE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1252: --- Attachment: hive-1252.3.patch a new patch added m.value is not null and m.value!='' > Task breaking bug when breaking after a filter operator > --- > > Key: HIVE-1252 > URL: https://issues.apache.org/jira/browse/HIVE-1252 > Project: Hadoop Hive > Issue Type: Bug >Reporter: He Yongqiang >Assignee: He Yongqiang > Fix For: 0.6.0 > > Attachments: hive-1252.1.patch, hive-1252.2.patch, hive-1252.3.patch > > > The plan for this query has a bug, > SELECT f.key, g.value FROM srcpart f JOIN srcpart m ON( f.key = m.key AND > f.ds='2008-04-08' AND m.ds='2008-04-08' ) JOIN srcpart g ON(g.value = m.value > AND g.ds='2008-04-08' AND m.ds='2008-04-08'); > The first file sink operator's table desc contains all columns. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1252) Task breaking bug when breaking after a filter operator
[ https://issues.apache.org/jira/browse/HIVE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846988#action_12846988 ] Namit Jain commented on HIVE-1252: -- Can you change the test to add: m.value is not null in the second join - it is painful to verify the results. > Task breaking bug when breaking after a filter operator > --- > > Key: HIVE-1252 > URL: https://issues.apache.org/jira/browse/HIVE-1252 > Project: Hadoop Hive > Issue Type: Bug >Reporter: He Yongqiang >Assignee: He Yongqiang > Fix For: 0.6.0 > > Attachments: hive-1252.1.patch, hive-1252.2.patch > > > The plan for this query has a bug, > SELECT f.key, g.value FROM srcpart f JOIN srcpart m ON( f.key = m.key AND > f.ds='2008-04-08' AND m.ds='2008-04-08' ) JOIN srcpart g ON(g.value = m.value > AND g.ds='2008-04-08' AND m.ds='2008-04-08'); > The first file sink operator's table desc contains all columns. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-867) Add add UDFs found in mysql
[ https://issues.apache.org/jira/browse/HIVE-867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846982#action_12846982 ] Edward Capriolo commented on HIVE-867: -- I am going to break these up into several Issues, encryption, constants, string, etc. This should avoid issues with one UDF holding all these others back. > Add add UDFs found in mysql > --- > > Key: HIVE-867 > URL: https://issues.apache.org/jira/browse/HIVE-867 > Project: Hadoop Hive > Issue Type: New Feature >Reporter: Edward Capriolo >Assignee: Edward Capriolo > Attachments: hive-867-1.diff, hive-867-10.diff, hive-867-2.diff, > hive-867-3.diff, hive-867-7.diff > > > Some UDF's that mysql has that hive does not. > atan > aes_decrypt > aes_encrypt > bit_and > bit_count > bit_length > bit_or > bit_xor > char_length > char > character_length > collation > compress > crc32 > encode > encrypt > format > greatest > in > inet_oton > inet_ntoa > match > md5 > oct > ord > pi > radians > sha1 _sha > sign > sleep > truncate -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
