Fwd: Problem in hive

-- Forwarded message -- From: Vikas Srivastava vikas.srivast...@one97.net Date: Tue, Aug 23, 2011 at 7:26 PM Subject: Problem in hive To: user@hive.apache.org HI team, i m facing this problem. show tables is running fine but when i run below query. hive select * from

Re: RE: Why a sql only use one map task?

What about your total Map Task Capacity? you may check it from http://your_jobtracker:50030/jobtracker.jsp 2011/8/24 Daniel,Wu hadoop...@163.com: I checked my setting, all are with the default value.So per the book of Hadoop the definitive guide, the split size should be 64M. And the file size

RE: Problem in hive

Hi, Can u post some more details like which version u r using and what sequence of queries u have executed. When I checked the trunk code this exception will come when getCols() returns null. Check u r metadata is in good state or not. Thanks Chinna Rao Lalam -- Forwarded

Re: Problem in hive

hey thanks for reply, i m using hadoop 0.20.2, and hive: 0.7.0 i have install hive on new server and making that use readonly , On Wed, Aug 24, 2011 at 4:05 PM, Chinna chinna...@huawei.com wrote: Hi, ** ** Can u post some more details like which version u r using and what

Re: Hive 0.7.1 authorization woes

I created the mysql database (with the simple create database command) and the remote metastore seemed to creat the mysql tables. Here's some grant information and what I see in the database: [hduser@aholmes-desktop conf]$ hive hive grant all to user hduser; OK Time taken: 0.334 seconds hive

Re:Re: RE: Why a sql only use one map task?

I pasted the inform I pasted blow, the map capacity is 6. And no matter how I set mapred.map.tasks, such as 3, it doesn't work, as it always use 1 map task (please see the completed job information). Cluster Summary (Heap Size is 16.81 MB/966.69 MB) Running Map TasksRunning Reduce

Re: Understanding distributed Hive server and Hive Metastore setup

On Wed, Aug 24, 2011 at 10:53 AM, Christian Kurz crk...@gmx.de wrote: Greetings, could somebody confirm/correct my understanding of a fully distributed Hive setup, please? My setup is as follows - *Java application using Hive JDBC driver *connects to - *hive --service

Re: Alter table Set Locations for all partitions

Thanks David, I will take a shot at this today. Best Bhupesh On Sun, Aug 21, 2011 at 6:51 PM, David Burley da...@geek.net wrote: Bhupesh, The following worked for us on a MySQL metastore: mysqldump metastore metastore-backup-`date +%Y%m%d`.sql mysql metastore UPDATE SDS SET

Re: Understanding distributed Hive server and Hive Metastore setup

Thanks for the quick reply, Edward I am not sure I got you: My HiveService has been started with hive.metastore.local=false. So shouldn't it use thrift instead of its own local Derby instance? Thanks, Christian Am 24.08.2011 um 19:33 schrieb Edward Capriolo edlinuxg...@gmail.com: On

Re: Hive 0.7.1 authorization woes

This is a bug. Will open a jira to fix this. and will backport it to 0.7.1. https://issues.apache.org/jira/browse/HIVE-2405 thanks for reporting this one! On Wed, Aug 24, 2011 at 6:25 AM, Alex Holmes grep.a...@gmail.com wrote: I created the mysql database (with the simple create database

Re: Hive 0.7.1 authorization woes

Thanks for opening a ticket. Table-level grants aren't working for me either (HIVE-2405 suggests that the bug is only related to global grants). hive set hive.security.authorization.enabled=false; hive CREATE TABLE pokes (foo INT, bar STRING); OK Time taken: 1.245 seconds hive LOAD DATA LOCAL

Re: Understanding distributed Hive server and Hive Metastore setup

On Wed, Aug 24, 2011 at 3:02 PM, Christian Kurz crk...@gmx.de wrote: Thanks for the quick reply, Edward I am not sure I got you: My HiveService has been started with hive.metastore.local=false. So shouldn't it use thrift instead of its own local Derby instance? Thanks, Christian Am

Re: Hive 0.7.1 authorization woes

I am using local metastore, and can not reproduce the problem. what message did you get when running local metastore? On Wed, Aug 24, 2011 at 1:58 PM, Alex Holmes grep.a...@gmail.com wrote: Thanks for opening a ticket. Table-level grants aren't working for me either (HIVE-2405 suggests that

Re: Hive 0.7.1 authorization woes

Authorization works for me with the local metastore. The remote metastore works with authorization turned off, but as soon as I turn it on and issue any commands I get these exceptions on the hive client. Could you also try the remote metastore please? I'm pretty sure that authorization does

Re: Hive 0.7.1 authorization woes

this is what i have tried with a remote metastore: set hive.security.authorization.enabled=false; hive drop table src2; OK Time taken: 1.002 seconds hive create table src2 (key int, value string); OK Time taken: 0.03 seconds hive set

Re: How to skip the malformatted records while loading data

One possibility is to filter out NULLs, something like following: hive select * from tb where id != NULL or pref != NULL or zip != NULL; This is not most efficient, but will work. 2011/8/18 XieXianshan xi...@cn.fujitsu.com Hi,everyone, Is there an option to ignore malformatted records while

Re: Re: RE: Why a sql only use one map task?

This may be because CombineHiveInputFormat is combining your splits in one map task. If you don't want that to happen, do: hive set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat 2011/8/24 Daniel,Wu hadoop...@163.com I pasted the inform I pasted blow, the map capacity is 6. And

RE: Re:RE: Why a sql only use one map task?

I think mapred.max.split.size is not set by default. The max split size is not the same as the HDFS block size. From: Daniel,Wu [mailto:hadoop...@163.com] Sent: Tuesday, August 23, 2011 11:44 PM To: user@hive.apache.org Subject: Re:RE: Why a sql only use one map task? I checked my setting, all

Re: Problem in hive

Vikas, Looks like your metadata is corrupted. Can you paste the output of following: hive describe formatted aircel_obd; Ashutosh On Wed, Aug 24, 2011 at 03:46, Vikas Srivastava vikas.srivast...@one97.netwrote: hey thanks for reply, i m using hadoop 0.20.2, and hive: 0.7.0 i have install

Re: Understanding distributed Hive server and Hive Metastore setup

Edward, Apart from recommended best practices what Christian is asking for is why HiveServer is still trying to interact with local db instance even after setting the config variables. AFAIK it should not. Christian, you found that out by looking at files opened by HiveServer jvm. Can you provide

Re: Maximum Nos of table in hive

Vikas, You are bounded by database used by metastore which usually is mysql. On a modern machine, mysql easily scales for few millions of rows, so I will expect that you should be able to create atleast those many tables. Thanks, Ashutosh On Wed, Aug 24, 2011 at 03:45, Chinna

Re: Maximum Nos of table in hive

On Wed, Aug 24, 2011 at 7:32 PM, Ashutosh Chauhan hashut...@apache.orgwrote: Vikas, You are bounded by database used by metastore which usually is mysql. On a modern machine, mysql easily scales for few millions of rows, so I will expect that you should be able to create atleast those many

Re: Local and remote metastores

Alex, Difference between the two is that in case of local metastore, hive client communicates directly to the mysql while in case of remote metastore, hive client first talks to a remote metastore which in turns talks to mysql. In terms of feature-set they are identical. Hope it helps, Ashutosh

Re: Local and remote metastores

Are there features that are only provided by the remote datastore (like authorization)? No. All features should be provided by both. In the case of authorization, the problem is that it did not get fully tested with a remote metastore. So you saw some small bugs when running authorization on a

Re: One Schema Per Partition? (Multiple schemas per table?)

Hey Tim, Hive does support different schema's for different partitions. If your data comes out garbled, that seems to be a bug then. In your case, is the following sequence of steps resemble what you did: a) create table tbl (id: int, name: string, level: int) partitioned by date; b) -- add