RE: How to install hive?

[Guillaume Polaert]

Hi,

You can also use a VM from Cloudera website : 
https://ccp.cloudera.com/display/SUPPORT/CDH+Downloads#CDHDownloads-CDH4PackagesandDownloads
 which includes Hive.

Guillaume Polaert | Cyrès

-Message d'origine-
De : Mohammad Tariq [mailto:donta...@gmail.com] 
Envoyé : mercredi 6 juin 2012 00:43
À : user@hive.apache.org
Objet : Re: How to install hive?

Hi Rafael,

   Just download Hive from apache and set all the variables properly, and you 
are good to go. For detailed help you can visit - 
"https://cwiki.apache.org/confluence/display/Hive/GettingStarted";...

And this is the official wikipage -
"https://cwiki.apache.org/confluence/display/Hive/Home";

Regards,
    Mohammad Tariq


On Wed, Jun 6, 2012 at 3:42 AM, Rafael Maffud Carlini  
wrote:
> Hello everyone, I develop a scientific research for my college, where 
> I conduct experiments involving hive and I wonder what is the easiest 
> way to install the hive.
>
> I've tried installing using claudera, but I had trouble picking up the 
> mirrors.
>
>  I'd rather install it using ubuntu or other debian distribution. In 
> my case, I need to work on Hadoop sigle-node cluster.
>
> I appreciate everyone's attention.

Re: subquery syntax error

[Igor Tatarinov]

The columns you ORDER BY must be SELECTed. If you don't want identifier2 in
the output, you will need to wrap your query with another SELECT.

Hope this helps.
igor
decide.com

On Tue, Jun 5, 2012 at 10:53 PM, Avdeev V. M.  wrote:

> Hello!
>
> I can't understand what's wrong.
>
>drop table if exists t0;
> create table t0 (short_descr string) STORED AS SEQUENCEFILE;
> from (
> select cast(2 as int) identifier2, short_descr area_name from t0) asdf
> select *--area_name
> order by identifier2 desc, area_name asc
> ;
>
> The query works fine but if I remove * then
>
> drop table if exists t0;
> create table t0 (short_descr string) STORED AS SEQUENCEFILE;
> from (
> select cast(2 as int) identifier2, short_descr area_name from t0) asdf
> select area_name
> order by identifier2 desc, area_name asc
> ;
>
> fails with error "FAILED: Error in semantic analysis: Line 6:9 Invalid
> table alias or column reference identifier2"
> The query
>
> drop table if exists t0;
> create table t0 (short_descr string) STORED AS SEQUENCEFILE;
> from (
> select short_descr area_name from t0) asdf
> select area_name
> order by area_name asc
> ;
>
> works fine too. What's wrong with 'no mapped' columns?
>
> Thanks!
>

subquery syntax error

[Avdeev V . M .]

Hello!

I can't understand what's wrong.

 

drop table if exists t0;
create table t0 (short_descr string) STORED AS SEQUENCEFILE;
from (
 select cast(2 as int) identifier2, short_descr area_name from t0) asdf
select *--area_name
order by identifier2 desc, area_name asc
;

The query works fine but if I remove * then

drop table if exists t0;
create table t0 (short_descr string) STORED AS SEQUENCEFILE;
from (
 select cast(2 as int) identifier2, short_descr area_name from t0) asdf
select area_name
order by identifier2 desc, area_name asc
;

fails with error "FAILED: Error in semantic analysis: Line 6:9 Invalid table 
alias or column reference identifier2"
The query

drop table if exists t0;
create table t0 (short_descr string) STORED AS SEQUENCEFILE;
from (
 select short_descr area_name from t0) asdf
select area_name
order by area_name asc
;

works fine too. What's wrong with 'no mapped' columns?

Thanks!

Re:Custom UDF in Python?

[王锋]

udfs need extends or implements the UDF class or other interface.so just java.

At 2012-06-06 12:21:03,"Saurabh S"  wrote:

Is it possible to write Hive UDFs in Python? I googled but didn't find 
anything. I would be happy with RTFM replies if you can give link to the manual.

Custom UDF in Python?

[Saurabh S]

Is it possible to write Hive UDFs in Python? I googled but didn't find 
anything. I would be happy with RTFM replies if you can give link to the manual.
  

Re: Multi-group-by select always scans entire table

[Mark Grover]

Hi Jan,
The quick answer is I don't know but may be someone else on the mailing
list does:-)

Looking at the wiki page for Lateral view(
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView),
there was a problem related to predicate pushdown on UDTF's (
https://issues.apache.org/jira/browse/HIVE-1056). However, that seemed to
have been fixed in Hive 0.6.0 so it shouldn't have any impact on you.

The fix for above ticket introduced a unit test (at
ql/src/test/results/clientpositive/lateral_view_ppd.q) that tests predicate
pushdown on UDTF's. Now, all the subsequent releases should have had that
test pass (otherwise they wouldn't have been released, I hope). The test
checks for a non-partition column for predicate pushdown. I wonder if it
makes a difference with a partition column being used.

Can you verify if your query with predicate pushdown enabled works as
expected with a non-partition column in the where clause? In that case, the
explain/explain extended output should be different from when predicate
pushdown is disabled. If predicate pushdown works for non-partition columns
but not for partition columns, please create a JIRA stating that predicate
pushdown on UDTF's doesn't work with partition columns.

If it doesn't work for both partition and non-partition columns, then
obviously Hive-1056 is not working for you. We can take it up on the
mailing list from there.

Thanks for your input, Jan.

Mark

On Tue, Jun 5, 2012 at 1:19 AM, Jan Dolinár  wrote:

>
>
> On Mon, Jun 4, 2012 at 7:20 PM, Mark Grover  wrote:
>
>> Hi Jan,
>> Glad you found something workable.
>>
>> What version of Hive are you using? Could you also please check what the
>> value of the property hive.optimize.ppd is for you?
>>
>> Thanks,
>> Mark
>>
>>
> Hi Mark,
>
> Thanks for reply. I'm using hive 0.7.1 distributed from Cloudera as
> cdh3u4. The property hive.optimize.ppd is set true, but I have tried to
> turn it off and it doesn't effect the behavior of the problematic query at
> all. Any other ideas? :-)
>
> Also could some of you good guys try to check this on hadoop 0.8 or newer?
> It would be nice to know if it is worth to go through all the hassle of
> upgrading or if it won't help. Also, if it is not fixed already, it might
> be good idea to report it as a bug.
>
> Jan
>

Question on Archive, Shims & “hive.archive.har.parentdir.settable” setting

[Kanna Karanam]

Hi Guys, While I am trying to understand the archive functionality/code in 
HIVE. I noticed that the default value of “hive.archive.har.parentdir.settable” 
setting is false. But shim is using this setting to pass one of the mandatory 
argument “–P” to HadoopArchives class.

Is there any place where we are setting this flag to run HIVE unit tests to 
pass this mandatory flag? I didn’t find it either in the hive-default.xml or 
hive-site.xml.

Thanks,
Kanna

Re: Need help running query over "yesterday". getting "No partition predicate found" errors

[Matthew Hooker]

Thanks, Mark.

I was hoping the expressions could be evaluated at query time, but
generating the query works just as well.

On Mon, Jun 4, 2012 at 7:42 PM, Mark Grover  wrote:

> Hi Matthew,
> unix_timestamp() is being executed on the Hive server in your case.
> Therefore, as your query progresses, the timestamp returned by your
> unix_timestamp() would change. Based on your usage, this would lead to
> unsuspecting problems when this queries runs across UTC midnight. In any
> case, I'd recommend that you get your Hive client code to get the present
> unix_timestamp and generate your partition pruning constant(s) on the
> client before you send your query to the server. When you do so, partition
> pruning will occur as expected.
>
> Mark
>
> - Original Message -
> From: "Matthew Hooker" 
> To: user@hive.apache.org
> Sent: Monday, June 4, 2012 8:01:26 PM
> Subject: Need help running query over "yesterday". getting "No partition
> predicate found" errors
>
> Hello,
>
>
> I'm trying to construct a query which will do some simple counts over a
> table of events, and insert them in to a summary table.
>
>
> The query I came up with looks something like
>
>
>
> INSERT OVERWRITE TABLE activity
> PARTITION(date_utc)
> select count(1) views,
> from impressions
>
> where dt >= date_sub(to_date(from_unixtime(unix_timestamp())),1)
> and dt < to_date(from_unixtime(unix_timestamp()))
> group by d_theme;
>
>
> This seemed to work fine, but I realized it was doing a full table scan,
> and not using the partitions in "views" appropriately.
>
>
> I set hive.mapred.mode=strict and the above query fails to run.
> If I change the expressions in the where clause to literals, it works as I
> would expect.
>
>
> Can anyone help me figure out how to do what I want?
>
>
>
>
> I put together a test script to illustrate my problem:
>
>
>
> set hive.mapred.mode=strict;
> set hive.optimize.ppd=true;
> DROP TABLE IF EXISTS test_where_expr;
> CREATE TABLE test_where_expr (
> ts int
> )
> PARTITIONED BY ( dt STRING );
> explain SELECT * from test_where_expr WHERE dt =
> to_date(date_sub(from_unixtime(unix_timestamp()),1));
> explain SELECT * from test_where_expr WHERE dt = '2012-06-01';
>
>
>
>
> This is the output:
>
>
>
> hive> set hive.mapred.mode=strict;
> hive> set hive.optimize.ppd=true;
> hive>
> > DROP TABLE IF EXISTS test_where_expr;
> OK
> Time taken: 3.405 seconds
> hive>
> > CREATE TABLE test_where_expr (
> > ts int
> > )
> > PARTITIONED BY ( dt STRING );
> OK
> Time taken: 0.189 seconds
> hive>
> > explain SELECT * from test_where_expr
> > WHERE dt = to_date(date_sub(from_unixtime(unix_timestamp()),1));
> FAILED: Error in semantic analysis: No partition predicate found for Alias
> "test_where_expr" Table "test_where_expr"
> hive>
> > explain SELECT * from test_where_expr
> > WHERE dt = '2012-06-01';
> OK
> ABSTRACT SYNTAX TREE:
> (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME test_where_expr)))
> (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT
> (TOK_SELEXPR TOK_ALLCOLREF)) (TOK_WHERE (= (TOK_TABLE_OR_COL dt)
> '2012-06-01'
>
>
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
>
>
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
>
>
>
>
> Time taken: 0.166 seconds
> hive> hadoop@ip-10-68-190-136:~$ hive --version
> Hive version 0.8.1.
> hadoop@ip-10-68-190-136:~$ hadoop -version
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
>
>
>
>
>
> Thanks,
> --Matthew Hooker
>

Re: How to install hive?

[Mohammad Tariq]

Hi Rafael,

   Just download Hive from apache and set all the variables properly,
and you are good to go. For detailed help you can visit -
"https://cwiki.apache.org/confluence/display/Hive/GettingStarted";...

And this is the official wikipage -
"https://cwiki.apache.org/confluence/display/Hive/Home";

Regards,
    Mohammad Tariq


On Wed, Jun 6, 2012 at 3:42 AM, Rafael Maffud Carlini
 wrote:
> Hello everyone, I develop a scientific research for my college, where
> I conduct experiments involving hive and I wonder what is the easiest
> way to install the hive.
>
> I've tried installing using claudera, but I had trouble picking up the 
> mirrors.
>
>  I'd rather install it using ubuntu or other debian distribution. In
> my case, I need to work on Hadoop sigle-node cluster.
>
> I appreciate everyone's attention.

How to install hive?

[Rafael Maffud Carlini]

Hello everyone, I develop a scientific research for my college, where
I conduct experiments involving hive and I wonder what is the easiest
way to install the hive.

I've tried installing using claudera, but I had trouble picking up the mirrors.

 I'd rather install it using ubuntu or other debian distribution. In
my case, I need to work on Hadoop sigle-node cluster.

I appreciate everyone's attention.

Re: Developing Hive UDF in eclipse

[Aniket Mokashi]

Put hive-exec*.jar in your eclipse classpath. (project properties-> java
build path -> libraries)

On Tue, Jun 5, 2012 at 8:52 AM, kulkarni.swar...@gmail.com <
kulkarni.swar...@gmail.com> wrote:

> Did you try this[1]? It had got me most of my way through the process.
>
> [1] https://cwiki.apache.org/Hive/gettingstarted-eclipsesetup.html
>
>
> On Tue, Jun 5, 2012 at 8:49 AM, Arun Prakash wrote:
>
>> Hi Friends,
>> I tried to develop udf for hive but i am getting package import error
>> in eclipse.
>>
>> import org.apache.hadoop.hive.ql.exec.UDF;
>>
>>
>> How to import hive package in eclipse?
>>
>>
>> Any inputs much appreciated.
>>
>>
>>
>> Best Regards
>>  Arun Prakash C.K
>>
>> Keep On Sharing Your Knowledge with Others
>>
>
>
>
> --
> Swarnim
>



-- 
"...:::Aniket:::... Quetzalco@tl"

Re: Error while Creating Table in Hive

[Babak Bastan]

no I'm not working on CDH.Is there a way to test if my Hadoop works fine or
not?

On Tue, Jun 5, 2012 at 9:55 PM, Bejoy KS  wrote:

> **
> Hi Babak
>
> You gotta follow those instructions in the apace site to set up hadoop
> from scratch and ensure that hdfs is working first. You should be able to
> read and write files to hdfs before you do your next steps.
>
> Are you on CDH or apache distribution of hadoop? If it is CDH there are
> detailed instructions on Cloudera web site.
>
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
> --
> *From: * Babak Bastan 
> *Date: *Tue, 5 Jun 2012 21:30:22 +0200
> *To: *
> *ReplyTo: * user@hive.apache.org
> *Subject: *Re: Error while Creating Table in Hive
>
> @Bejoy: I set the fs.default.name in the core-site.xml and I did all of
> thing that was mentioned in the reference but no effect
>
> On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan  wrote:
>
>> Ok sorry but that was my Mistake .I thought it works but no.
>> I wrote the command without ; and then I think It works but with ; at the
>> end of command
>>
>> CREATE TABLE pokes (foo INT, bar STRING);
>>
>> does'nt work
>>
>>
>> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv <
>> dwivedishash...@gmail.com> wrote:
>>
>>> inside configuration. all properties will be inside the configuration
>>> tags
>>>
>>>
>>> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan wrote:
>>>
 Thank you so much my friend your idee works fine(no error) you are the
 best :)


 On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan wrote:

> It must be inside the  or outside this?
>
>
> On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv <
> dwivedishash...@gmail.com> wrote:
>
>> It will be inside hive/conf
>>
>>
>> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan wrote:
>>
>>> Thanks sShashwat, and where is this hive-site.xml
>>>
>>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
>>> dwivedishash...@gmail.com> wrote:
>>>
 set

 hive.metastore.warehouse.dir in hive-site.xml

 
   hive.metastore.local
   true
 

 hive.metastore.warehouse.dir
/home//hivefolder
location of default database for the
 warehouse



 On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan 
 wrote:

> Hello Experts ,
>
> I'm new in Hive .When try to create a test Table in Hive I get an
> error.I want to run this command:
> *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
> but this error occured:
> FAILED: Error in metadata: MetaException(message:Got exception:
> java.io.FileNotFoundException File file:/user/hive/warehouse/test 
> does not
> exist.)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> How can I solve this Problem?
> Thank you so much
>



 --


 ∞
 Shashwat Shriparv



>>>
>>
>>
>> --
>>
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>>
>

>>>
>>>
>>> --
>>>
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>>
>>
>

Re: Error while Creating Table in Hive

[Bejoy KS]

Hi Babak

You gotta follow those instructions in the apace site to set up hadoop from 
scratch and ensure that hdfs is working first. You should be able to read and 
write files to hdfs before you do your next steps.

Are you on CDH or apache distribution of hadoop? If it is CDH there are 
detailed instructions on Cloudera web site.


Regards
Bejoy KS

Sent from handheld, please excuse typos.

-Original Message-
From: Babak Bastan 
Date: Tue, 5 Jun 2012 21:30:22 
To: 
Reply-To: user@hive.apache.org
Subject: Re: Error while Creating Table in Hive

@Bejoy: I set the fs.default.name in the core-site.xml and I did all of
thing that was mentioned in the reference but no effect

On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan  wrote:

> Ok sorry but that was my Mistake .I thought it works but no.
> I wrote the command without ; and then I think It works but with ; at the
> end of command
>
> CREATE TABLE pokes (foo INT, bar STRING);
>
> does'nt work
>
>
> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv <
> dwivedishash...@gmail.com> wrote:
>
>> inside configuration. all properties will be inside the configuration tags
>>
>>
>> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan  wrote:
>>
>>> Thank you so much my friend your idee works fine(no error) you are the
>>> best :)
>>>
>>>
>>> On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan  wrote:
>>>
 It must be inside the  or outside this?


 On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv <
 dwivedishash...@gmail.com> wrote:

> It will be inside hive/conf
>
>
> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan wrote:
>
>> Thanks sShashwat, and where is this hive-site.xml
>>
>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
>> dwivedishash...@gmail.com> wrote:
>>
>>> set
>>>
>>> hive.metastore.warehouse.dir in hive-site.xml
>>>
>>> 
>>>   hive.metastore.local
>>>   true
>>> 
>>>
>>> hive.metastore.warehouse.dir
>>>/home//hivefolder
>>>location of default database for the
>>> warehouse
>>>
>>>
>>>
>>> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan wrote:
>>>
 Hello Experts ,

 I'm new in Hive .When try to create a test Table in Hive I get an
 error.I want to run this command:
 *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
 but this error occured:
 FAILED: Error in metadata: MetaException(message:Got exception:
 java.io.FileNotFoundException File file:/user/hive/warehouse/test does 
 not
 exist.)
 FAILED: Execution Error, return code 1 from
 org.apache.hadoop.hive.ql.exec.DDLTask
 How can I solve this Problem?
 Thank you so much

>>>
>>>
>>>
>>> --
>>>
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>>
>>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>

>>>
>>
>>
>> --
>>
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>>
>

Re: Error while Creating Table in Hive

[Babak Bastan]

@Bejoy: I set the fs.default.name in the core-site.xml and I did all of
thing that was mentioned in the reference but no effect

On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan  wrote:

> Ok sorry but that was my Mistake .I thought it works but no.
> I wrote the command without ; and then I think It works but with ; at the
> end of command
>
> CREATE TABLE pokes (foo INT, bar STRING);
>
> does'nt work
>
>
> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv <
> dwivedishash...@gmail.com> wrote:
>
>> inside configuration. all properties will be inside the configuration tags
>>
>>
>> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan  wrote:
>>
>>> Thank you so much my friend your idee works fine(no error) you are the
>>> best :)
>>>
>>>
>>> On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan  wrote:
>>>
 It must be inside the  or outside this?


 On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv <
 dwivedishash...@gmail.com> wrote:

> It will be inside hive/conf
>
>
> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan wrote:
>
>> Thanks sShashwat, and where is this hive-site.xml
>>
>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
>> dwivedishash...@gmail.com> wrote:
>>
>>> set
>>>
>>> hive.metastore.warehouse.dir in hive-site.xml
>>>
>>> 
>>>   hive.metastore.local
>>>   true
>>> 
>>>
>>> hive.metastore.warehouse.dir
>>>/home//hivefolder
>>>location of default database for the
>>> warehouse
>>>
>>>
>>>
>>> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan wrote:
>>>
 Hello Experts ,

 I'm new in Hive .When try to create a test Table in Hive I get an
 error.I want to run this command:
 *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
 but this error occured:
 FAILED: Error in metadata: MetaException(message:Got exception:
 java.io.FileNotFoundException File file:/user/hive/warehouse/test does 
 not
 exist.)
 FAILED: Execution Error, return code 1 from
 org.apache.hadoop.hive.ql.exec.DDLTask
 How can I solve this Problem?
 Thank you so much

>>>
>>>
>>>
>>> --
>>>
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>>
>>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>

>>>
>>
>>
>> --
>>
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>>
>

Re: Error while Creating Table in Hive

[Babak Bastan]

Ok sorry but that was my Mistake .I thought it works but no.
I wrote the command without ; and then I think It works but with ; at the
end of command

CREATE TABLE pokes (foo INT, bar STRING);

does'nt work


On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv  wrote:

> inside configuration. all properties will be inside the configuration tags
>
>
> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan  wrote:
>
>> Thank you so much my friend your idee works fine(no error) you are the
>> best :)
>>
>>
>> On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan  wrote:
>>
>>> It must be inside the  or outside this?
>>>
>>>
>>> On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv <
>>> dwivedishash...@gmail.com> wrote:
>>>
 It will be inside hive/conf


 On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan wrote:

> Thanks sShashwat, and where is this hive-site.xml
>
> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
> dwivedishash...@gmail.com> wrote:
>
>> set
>>
>> hive.metastore.warehouse.dir in hive-site.xml
>>
>> 
>>   hive.metastore.local
>>   true
>> 
>>
>> hive.metastore.warehouse.dir
>>/home//hivefolder
>>location of default database for the
>> warehouse
>>
>>
>>
>> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan wrote:
>>
>>> Hello Experts ,
>>>
>>> I'm new in Hive .When try to create a test Table in Hive I get an
>>> error.I want to run this command:
>>> *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
>>> but this error occured:
>>> FAILED: Error in metadata: MetaException(message:Got exception:
>>> java.io.FileNotFoundException File file:/user/hive/warehouse/test does 
>>> not
>>> exist.)
>>> FAILED: Execution Error, return code 1 from
>>> org.apache.hadoop.hive.ql.exec.DDLTask
>>> How can I solve this Problem?
>>> Thank you so much
>>>
>>
>>
>>
>> --
>>
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>>
>


 --


 ∞
 Shashwat Shriparv



>>>
>>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>

Re: Error while Creating Table in Hive

[shashwat shriparv]

inside configuration. all properties will be inside the configuration tags

On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan  wrote:

> Thank you so much my friend your idee works fine(no error) you are the
> best :)
>
>
> On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan  wrote:
>
>> It must be inside the  or outside this?
>>
>>
>> On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv <
>> dwivedishash...@gmail.com> wrote:
>>
>>> It will be inside hive/conf
>>>
>>>
>>> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan wrote:
>>>
 Thanks sShashwat, and where is this hive-site.xml

 On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
 dwivedishash...@gmail.com> wrote:

> set
>
> hive.metastore.warehouse.dir in hive-site.xml
>
> 
>   hive.metastore.local
>   true
> 
>
> hive.metastore.warehouse.dir
>/home//hivefolder
>location of default database for the
> warehouse
>
>
>
> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan wrote:
>
>> Hello Experts ,
>>
>> I'm new in Hive .When try to create a test Table in Hive I get an
>> error.I want to run this command:
>> *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
>> but this error occured:
>> FAILED: Error in metadata: MetaException(message:Got exception:
>> java.io.FileNotFoundException File file:/user/hive/warehouse/test does 
>> not
>> exist.)
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.DDLTask
>> How can I solve this Problem?
>> Thank you so much
>>
>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>

>>>
>>>
>>> --
>>>
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>>
>>
>


-- 


∞
Shashwat Shriparv

Re: Error while Creating Table in Hive

[Babak Bastan]

Thank you so much my friend your idee works fine(no error) you are the best
:)

On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan  wrote:

> It must be inside the  or outside this?
>
>
> On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv <
> dwivedishash...@gmail.com> wrote:
>
>> It will be inside hive/conf
>>
>>
>> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan  wrote:
>>
>>> Thanks sShashwat, and where is this hive-site.xml
>>>
>>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
>>> dwivedishash...@gmail.com> wrote:
>>>
 set

 hive.metastore.warehouse.dir in hive-site.xml

 
   hive.metastore.local
   true
 

 hive.metastore.warehouse.dir
/home//hivefolder
location of default database for the
 warehouse



 On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan wrote:

> Hello Experts ,
>
> I'm new in Hive .When try to create a test Table in Hive I get an
> error.I want to run this command:
> *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
> but this error occured:
> FAILED: Error in metadata: MetaException(message:Got exception:
> java.io.FileNotFoundException File file:/user/hive/warehouse/test does not
> exist.)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> How can I solve this Problem?
> Thank you so much
>



 --


 ∞
 Shashwat Shriparv



>>>
>>
>>
>> --
>>
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>>
>

Re: Error while Creating Table in Hive

[Babak Bastan]

It must be inside the  or outside this?

On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv  wrote:

> It will be inside hive/conf
>
>
> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan  wrote:
>
>> Thanks sShashwat, and where is this hive-site.xml
>>
>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
>> dwivedishash...@gmail.com> wrote:
>>
>>> set
>>>
>>> hive.metastore.warehouse.dir in hive-site.xml
>>>
>>> 
>>>   hive.metastore.local
>>>   true
>>> 
>>>
>>> hive.metastore.warehouse.dir
>>>/home//hivefolder
>>>location of default database for the
>>> warehouse
>>>
>>>
>>>
>>> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan wrote:
>>>
 Hello Experts ,

 I'm new in Hive .When try to create a test Table in Hive I get an
 error.I want to run this command:
 *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
 but this error occured:
 FAILED: Error in metadata: MetaException(message:Got exception:
 java.io.FileNotFoundException File file:/user/hive/warehouse/test does not
 exist.)
 FAILED: Execution Error, return code 1 from
 org.apache.hadoop.hive.ql.exec.DDLTask
 How can I solve this Problem?
 Thank you so much

>>>
>>>
>>>
>>> --
>>>
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>>
>>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>

Re: Error while Creating Table in Hive

[shashwat shriparv]

/etc/hive/conf/hive-site.xml

check out at above folder there you make changes

for help i am sending you link

https://ccp.cloudera.com/display/CDHDOC/Hive+Installation

On Tue, Jun 5, 2012 at 11:45 PM, shashwat shriparv <
dwivedishash...@gmail.com> wrote:

> It will be inside hive/conf
>
>
> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan  wrote:
>
>> Thanks sShashwat, and where is this hive-site.xml
>>
>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
>> dwivedishash...@gmail.com> wrote:
>>
>>> set
>>>
>>> hive.metastore.warehouse.dir in hive-site.xml
>>>
>>> 
>>>   hive.metastore.local
>>>   true
>>> 
>>>
>>> hive.metastore.warehouse.dir
>>>/home//hivefolder
>>>location of default database for the
>>> warehouse
>>>
>>>
>>>
>>> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan wrote:
>>>
 Hello Experts ,

 I'm new in Hive .When try to create a test Table in Hive I get an
 error.I want to run this command:
 *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
 but this error occured:
 FAILED: Error in metadata: MetaException(message:Got exception:
 java.io.FileNotFoundException File file:/user/hive/warehouse/test does not
 exist.)
 FAILED: Execution Error, return code 1 from
 org.apache.hadoop.hive.ql.exec.DDLTask
 How can I solve this Problem?
 Thank you so much

>>>
>>>
>>>
>>> --
>>>
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>>
>>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>


-- 


∞
Shashwat Shriparv

Re: Error while Creating Table in Hive

[shashwat shriparv]

It will be inside hive/conf

On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan  wrote:

> Thanks sShashwat, and where is this hive-site.xml
>
> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
> dwivedishash...@gmail.com> wrote:
>
>> set
>>
>> hive.metastore.warehouse.dir in hive-site.xml
>>
>> 
>>   hive.metastore.local
>>   true
>> 
>>
>> hive.metastore.warehouse.dir
>>/home//hivefolder
>>location of default database for the
>> warehouse
>>
>>
>>
>> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan  wrote:
>>
>>> Hello Experts ,
>>>
>>> I'm new in Hive .When try to create a test Table in Hive I get an
>>> error.I want to run this command:
>>> *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
>>> but this error occured:
>>> FAILED: Error in metadata: MetaException(message:Got exception:
>>> java.io.FileNotFoundException File file:/user/hive/warehouse/test does not
>>> exist.)
>>> FAILED: Execution Error, return code 1 from
>>> org.apache.hadoop.hive.ql.exec.DDLTask
>>> How can I solve this Problem?
>>> Thank you so much
>>>
>>
>>
>>
>> --
>>
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>>
>


-- 


∞
Shashwat Shriparv

Re: Error while Creating Table in Hive

[Babak Bastan]

Thanks sShashwat, and where is this hive-site.xml

On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv  wrote:

> set
>
> hive.metastore.warehouse.dir in hive-site.xml
>
> 
>   hive.metastore.local
>   true
> 
>
> hive.metastore.warehouse.dir
>/home//hivefolder
>location of default database for the
> warehouse
>
>
>
> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan  wrote:
>
>> Hello Experts ,
>>
>> I'm new in Hive .When try to create a test Table in Hive I get an error.I
>> want to run this command:
>> *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
>> but this error occured:
>> FAILED: Error in metadata: MetaException(message:Got exception:
>> java.io.FileNotFoundException File file:/user/hive/warehouse/test does not
>> exist.)
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.DDLTask
>> How can I solve this Problem?
>> Thank you so much
>>
>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>

Re: Error while Creating Table in Hive

[shashwat shriparv]

set

hive.metastore.warehouse.dir in hive-site.xml


  hive.metastore.local
  true


hive.metastore.warehouse.dir
   /home//hivefolder
   location of default database for the
warehouse
   


On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan  wrote:

> Hello Experts ,
>
> I'm new in Hive .When try to create a test Table in Hive I get an error.I
> want to run this command:
> *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
> but this error occured:
> FAILED: Error in metadata: MetaException(message:Got exception:
> java.io.FileNotFoundException File file:/user/hive/warehouse/test does not
> exist.)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> How can I solve this Problem?
> Thank you so much
>



-- 


∞
Shashwat Shriparv

Re: Error while Creating Table in Hive

[Bejoy Ks]

HI Babak

It looks like a hadoop configuration problem for me. Have you configured your 
'fs.default.name'  in core-site.xl to point to hdfs:// instead of file:/// ? If 
not that is likely to be the issue. Also if you are usinh hdfs for first time 
there are other steps like formatting namenode and all to be performed.

http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html

Ensure your hadoop installation is fully working before you get on to hive.



Regards
Bejoy KS






 From: Babak Bastan 
To: user@hive.apache.org 
Sent: Tuesday, June 5, 2012 10:43 PM
Subject: Error while Creating Table in Hive
 

Hello Experts ,

I'm new in Hive .When try to create a test Table in Hive I get an error.I want 
to run this command:
CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);
but this error occured:
FAILED: Error in metadata: MetaException(message:Got exception: 
java.io.FileNotFoundException File file:/user/hive/warehouse/test does not 
exist.)
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask
How can I solve this Problem?
Thank you so much

Re: Twitter Data analyse with HIVE

[Bejoy Ks]

Hi Babak

Looks like your hadoop is not configured correctly. The list gives me a pulse 
that it is showing lfs rather than hdfs.  Have you configured your 
'fs.default.name'  in core-site.xl to point to hdfs:// instead of file:/// . 
You may need to revisit your hadoop setup. Try out the book I recommend, It is 
kick ass and will resolve all your queries.

Regards,
Bejoy KS






 From: Babak Bastan 
To: user@hive.apache.org 
Sent: Tuesday, June 5, 2012 10:17 PM
Subject: Re: Twitter Data analyse with HIVE
 

Thank you Bejoy for your complete answer :)

if I run this command:

hadoop fs -ls /
I get this results:
drwxr-xr-x   - root root       4096 2011-04-26 01:06 /var
drwxrwxrwx   - root root       4096 2012-06-05 18:38 /tmp
drwxr-xr-x   - root root      12288 2012-06-05 17:44 /etc
-rw-r--r--   1 root root   12809911 2012-06-02 09:57 /initrd.img
drwxr-xr-x   - root root       4340 2012-06-05 17:42 /dev
drwxr-xr-x   - root root       4096 2012-06-02 09:57 /boot
drwxr-xr-x   - root root       4096 2011-04-26 00:50 /srv
drwxr-xr-x   - root root       4096 2012-06-01 11:45 /user
-rw-r--r--   1 root root   12832710 2012-06-02 09:56 /initrd.img.old
drwxr-xr-x   - root root       4096 2012-06-02 09:52 /lib
drwxr-xr-x   - root root       4096 2012-06-05 12:52 /media
drwxrwxrwx   - root root      12288 2012-06-02 08:13 /host
-rw---   1 root root    4654608 2011-06-28 23:30 /vmlinuz.old
drwxr-xr-x   - root root       4096 2012-06-02 09:54 /sbin
drwxr-xr-x   - root root       4096 2012-06-01 11:36 /babak
dr-xr-xr-x   - root root          0 2012-06-05 12:22 /proc
drwxr-xr-x   - root root       4096 2012-05-31 22:03 /Downloads

What does the first column mean? I tried to make a dir in Downloads 

hadoop fs -mkdir /Downloads/TwitterData

but no success and the system said:

mkdir: failed to create /Downloads/TwitterData

in Downloads I can't make a directory? 


On Tue, Jun 5, 2012 at 6:13 PM, Sonal Goyal  wrote:

Lfs means local file system. 
>
>
>Hadoop fs -copyFromLocal will help to copy data from your local file system to 
>the Hadoop distributed file system. Not sure what kind of cluster setup you 
>have, are you running in local or pseudo distributed mode?
>
>
>Here is a link to get you started on 
>hivehttps://cwiki.apache.org/confluence/display/Hive/GettingStarted
>
>
>You can specifically look for 'load data local in path' for using the local 
>file system.
>
>
>
>And here is a link specifically regarding tweets.
>
>http://www.cloudera.com/blog/2010/12/hadoop-world-2010-tweet-analysis/
>
>Sent from my iPad
>
>On 05-Jun-2012, at 9:27 PM, Babak Bastan  wrote:
>
>
>Thank you for your answer
>>location of file in lfs
>>
>>That means the location of my *txt file on my computer ? and I have no 
>>destination address in hdfs,where can I get this location?
>>could you please write an example?
>>
>>
>>On Tue, Jun 5, 2012 at 4:29 PM, Bejoy Ks  wrote:
>>
>>Hi Babak
>>>
>>>
>>>There isn't anything called hdfs files. Hdfs is just a file system that can 
>>>store any type of file. You just need to transfer your file from lfs to hdfs 
>>>and the following command helps you out for that
>>>
>>>
>>>hadoop fs -copyFromLocal  >>hdfs>
>>>
>>>
>>>Regards
>>>Bejoy KS
>>>
>>>
>>>
>>>
>>> From: Babak Bastan 
>>>To: user@hive.apache.org 
>>>Sent: Tuesday, June 5, 2012 7:54 PM
>>>Subject: Re: Twitter Data analyse with HIVE
>>> 
>>>
>>>ok, no difference for me records in a line or not
>>> 2009-06-08 21:49:37 - http://twitter.com/evionblablabla- I think data 
>>>mining is awesome!
>>>2009-06-08 21:49:37 - http://twitter.com/hyungjinbliblibli -  I don’t think 
>>>so. I don’t like data mining
>>>
>>>
>>>
>>>How can I do that.I think that I should change my text file to hdfs 
>>>file,correct? how can I do this one?
>>>Sorry I'm very new in this field :(
>>>
>>>
>>>On Tue, Jun 5, 2012 at 4:07 PM, Edward Capriolo  
>>>wrote:
>>>
>>>If you get output onto a single line it will be much easier for hive to 
>>>process.


On Tue, Jun 5, 2012 at 5:20 AM, Babak Bastan  wrote:
> Hi experts
>
> I'm very new in Hive and Hadoop and I want to create a very simple demo to
> analyse sample twitts like this:
>
> T 2009-06-08 21:49:37
> U http://twitter.com/evion
> W I think data mining is awesome!
>
> T 2009-06-08 21:49:37
> U http://twitter.com/hyungjin
> W I don’t think so. I don’t like data mining
> 
> Generally is it possible to do that?
> but I don't know exactly from which point should I strat.Do you know any
> simple and clear reference to do this job? or  would you please inform me
> (not in detail) what should I do?
>
> Thank you very much for your helps
> Babak

>>>
>>>
>>>
>>

Error while Creating Table in Hive

[Babak Bastan]

Hello Experts ,

I'm new in Hive .When try to create a test Table in Hive I get an error.I
want to run this command:
*CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
but this error occured:
FAILED: Error in metadata: MetaException(message:Got exception:
java.io.FileNotFoundException File file:/user/hive/warehouse/test does not
exist.)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
How can I solve this Problem?
Thank you so much

Hive /w hbase-0.94.0

[Sean McNamara]

We're currently using hive-0.8.1 with hbase-0.92.0 and looking to rev both.  
Has anyone tried using hive with hbase-0.94.0?  Is there a particular version 
of hive that we would need to use?  The newer versions of hive look like they 
still have the 0.92 jars.

Thanks

Sean

Re: Twitter Data analyse with HIVE

[Babak Bastan]

Thank you Bejoy for your complete answer :)

if I run this command:

hadoop fs -ls /
I get this results:
drwxr-xr-x   - root root   4096 2011-04-26 01:06 /var
drwxrwxrwx   - root root   4096 2012-06-05 18:38 /tmp
drwxr-xr-x   - root root  12288 2012-06-05 17:44 /etc
-rw-r--r--   1 root root   12809911 2012-06-02 09:57 /initrd.img
drwxr-xr-x   - root root   4340 2012-06-05 17:42 /dev
drwxr-xr-x   - root root   4096 2012-06-02 09:57 /boot
drwxr-xr-x   - root root   4096 2011-04-26 00:50 /srv
drwxr-xr-x   - root root   4096 2012-06-01 11:45 /user
-rw-r--r--   1 root root   12832710 2012-06-02 09:56 /initrd.img.old
drwxr-xr-x   - root root   4096 2012-06-02 09:52 /lib
drwxr-xr-x   - root root   4096 2012-06-05 12:52 /media
drwxrwxrwx   - root root  12288 2012-06-02 08:13 /host
-rw---   1 root root4654608 2011-06-28 23:30 /vmlinuz.old
drwxr-xr-x   - root root   4096 2012-06-02 09:54 /sbin
drwxr-xr-x   - root root   4096 2012-06-01 11:36 /babak
dr-xr-xr-x   - root root  0 2012-06-05 12:22 /proc
*drwxr-xr-x   - root root   4096 2012-05-31 22:03 /Downloads*
*
*
What does the first column mean? I tried to make a dir in *Downloads*

hadoop fs -mkdir /Downloads/TwitterData

but no success and the system said:

mkdir: failed to create /Downloads/TwitterData

in Downloads I can't make a directory?


On Tue, Jun 5, 2012 at 6:13 PM, Sonal Goyal  wrote:

> Lfs means local file system.
>
> Hadoop fs -copyFromLocal will help to copy data from your local file
> system to the Hadoop distributed file system. Not sure what kind of cluster
> setup you have, are you running in local or pseudo distributed mode?
>
> Here is a link to get you started on hive
> https://cwiki.apache.org/confluence/display/Hive/GettingStarted
>
> You can specifically look for 'load data local in path' for using the
> local file system.
>
> And here is a link specifically regarding tweets.
>
> http://www.cloudera.com/blog/2010/12/hadoop-world-2010-tweet-analysis/
>
> Sent from my iPad
>
> On 05-Jun-2012, at 9:27 PM, Babak Bastan  wrote:
>
> Thank you for your answer
> location of file in lfs
> That means the location of my *txt file on my computer ? and I have no
> destination address in hdfs,where can I get this location?
> could you please write an example?
>
> On Tue, Jun 5, 2012 at 4:29 PM, Bejoy Ks  wrote:
>
>> Hi Babak
>>
>> There isn't anything called hdfs files. Hdfs is just a file system that
>> can store any type of file. You just need to transfer your file from lfs to
>> hdfs and the following command helps you out for that
>>
>> hadoop fs -copyFromLocal  > in hdfs>
>>
>> Regards
>> Bejoy KS
>>
>>   --
>> *From:* Babak Bastan 
>> *To:* user@hive.apache.org
>> *Sent:* Tuesday, June 5, 2012 7:54 PM
>> *Subject:* Re: Twitter Data analyse with HIVE
>>
>> ok, no difference for me records in a line or not
>>  2009-06-08 21:49:37 - http://twitter.com/evionblablabla- I think data
>> mining is awesome!
>> 2009-06-08 21:49:37 - http://twitter.com/ 
>> bliblibli
>> -  I don’t think so. I don’t like data mining
>> 
>>
>> How can I do that.I think that I should change my text file to hdfs
>> file,correct? how can I do this one?
>> Sorry I'm very new in this field :(
>>
>> On Tue, Jun 5, 2012 at 4:07 PM, Edward Capriolo wrote:
>>
>> If you get output onto a single line it will be much easier for hive to
>> process.
>>
>> On Tue, Jun 5, 2012 at 5:20 AM, Babak Bastan  wrote:
>> > Hi experts
>> >
>> > I'm very new in Hive and Hadoop and I want to create a very simple demo
>> to
>> > analyse sample twitts like this:
>> >
>> > T 2009-06-08 21:49:37
>> > U http://twitter.com/evion
>> > W I think data mining is awesome!
>> >
>> > T 2009-06-08 21:49:37
>> > U http://twitter.com/hyungjin
>> > W I don’t think so. I don’t like data mining
>> > 
>> > Generally is it possible to do that?
>> > but I don't know exactly from which point should I strat.Do you know any
>> > simple and clear reference to do this job? or  would you please inform
>> me
>> > (not in detail) what should I do?
>> >
>> > Thank you very much for your helps
>> > Babak
>>
>>
>>
>>
>>
>

Re: Twitter Data analyse with HIVE

[Sonal Goyal]

Lfs means local file system. 

Hadoop fs -copyFromLocal will help to copy data from your local file system to 
the Hadoop distributed file system. Not sure what kind of cluster setup you 
have, are you running in local or pseudo distributed mode?

Here is a link to get you started on hive
https://cwiki.apache.org/confluence/display/Hive/GettingStarted

You can specifically look for 'load data local in path' for using the local 
file system.

And here is a link specifically regarding tweets.

http://www.cloudera.com/blog/2010/12/hadoop-world-2010-tweet-analysis/

Sent from my iPad

On 05-Jun-2012, at 9:27 PM, Babak Bastan  wrote:

> Thank you for your answer
> location of file in lfs
> That means the location of my *txt file on my computer ? and I have no 
> destination address in hdfs,where can I get this location?
> could you please write an example?
> 
> On Tue, Jun 5, 2012 at 4:29 PM, Bejoy Ks  wrote:
> Hi Babak
> 
> There isn't anything called hdfs files. Hdfs is just a file system that can 
> store any type of file. You just need to transfer your file from lfs to hdfs 
> and the following command helps you out for that
> 
> hadoop fs -copyFromLocal   hdfs>
> 
> Regards
> Bejoy KS
> 
> From: Babak Bastan 
> To: user@hive.apache.org 
> Sent: Tuesday, June 5, 2012 7:54 PM
> Subject: Re: Twitter Data analyse with HIVE
> 
> ok, no difference for me records in a line or not
>  2009-06-08 21:49:37 - http://twitter.com/evionblablabla- I think data mining 
> is awesome!
> 2009-06-08 21:49:37 - http://twitter.com/bliblibli -  I don’t think so. I 
> don’t like data mining
> 
> 
> How can I do that.I think that I should change my text file to hdfs 
> file,correct? how can I do this one?
> Sorry I'm very new in this field :(
> 
> On Tue, Jun 5, 2012 at 4:07 PM, Edward Capriolo  wrote:
> If you get output onto a single line it will be much easier for hive to 
> process.
> 
> On Tue, Jun 5, 2012 at 5:20 AM, Babak Bastan  wrote:
> > Hi experts
> >
> > I'm very new in Hive and Hadoop and I want to create a very simple demo to
> > analyse sample twitts like this:
> >
> > T 2009-06-08 21:49:37
> > U http://twitter.com/evion
> > W I think data mining is awesome!
> >
> > T 2009-06-08 21:49:37
> > U http://twitter.com/hyungjin
> > W I don’t think so. I don’t like data mining
> > 
> > Generally is it possible to do that?
> > but I don't know exactly from which point should I strat.Do you know any
> > simple and clear reference to do this job? or  would you please inform me
> > (not in detail) what should I do?
> >
> > Thank you very much for your helps
> > Babak
> 
> 
> 
> 

Re: Twitter Data analyse with HIVE

[Bejoy Ks]

Hi Babak

In very simple terms, you have a set of machines on linux OS. You may have your 
files in that linux file system . From this linux file system you need to copy 
to hdfs. In hdfs create a directory like you do in normal systems which ought 
to be your destination directory. copy the file/dir from this linux file system 
to hdfs.


to list the contents in hdfs
hadoop fs -ls /

make a dir in hdfs
hadoop fs -mkdir /userdata/user/twitterData

copy from lfs to hdfs
hadoop fs -copyFromLocal /home/user/tweetData /userdata/user/twitterData 

It'll be much better if you can start off your hive experiments after reading 
'Hadoop - The Definitive Guide' by Tom White, It is a amazing  resource for 
hadoop starters.


Regards
Bejoy KS




 From: Babak Bastan 
To: user@hive.apache.org; Bejoy Ks  
Sent: Tuesday, June 5, 2012 9:27 PM
Subject: Re: Twitter Data analyse with HIVE
 

Thank you for your answer
location of file in lfs

That means the location of my *txt file on my computer ? and I have no 
destination address in hdfs,where can I get this location?
could you please write an example?


On Tue, Jun 5, 2012 at 4:29 PM, Bejoy Ks  wrote:

Hi Babak
>
>
>There isn't anything called hdfs files. Hdfs is just a file system that can 
>store any type of file. You just need to transfer your file from lfs to hdfs 
>and the following command helps you out for that
>
>
>hadoop fs -copyFromLocal  hdfs>
>
>
>Regards
>Bejoy KS
>
>
>
>
> From: Babak Bastan 
>To: user@hive.apache.org 
>Sent: Tuesday, June 5, 2012 7:54 PM
>Subject: Re: Twitter Data analyse with HIVE
> 
>
>ok, no difference for me records in a line or not
> 2009-06-08 21:49:37 - http://twitter.com/evionblablabla- I think data mining 
>is awesome!
>2009-06-08 21:49:37 - http://twitter.com/hyungjinbliblibli -  I don’t think 
>so. I don’t like data mining
>
>
>
>How can I do that.I think that I should change my text file to hdfs 
>file,correct? how can I do this one?
>Sorry I'm very new in this field :(
>
>
>On Tue, Jun 5, 2012 at 4:07 PM, Edward Capriolo  wrote:
>
>If you get output onto a single line it will be much easier for hive to 
>process.
>>
>>
>>On Tue, Jun 5, 2012 at 5:20 AM, Babak Bastan  wrote:
>>> Hi experts
>>>
>>> I'm very new in Hive and Hadoop and I want to create a very simple demo to
>>> analyse sample twitts like this:
>>>
>>> T 2009-06-08 21:49:37
>>> U http://twitter.com/evion
>>> W I think data mining is awesome!
>>>
>>> T 2009-06-08 21:49:37
>>> U http://twitter.com/hyungjin
>>> W I don’t think so. I don’t like data mining
>>> 
>>> Generally is it possible to do that?
>>> but I don't know exactly from which point should I strat.Do you know any
>>> simple and clear reference to do this job? or  would you please inform me
>>> (not in detail) what should I do?
>>>
>>> Thank you very much for your helps
>>> Babak
>>
>
>
>

RE: Twitter Data analyse with HIVE

[Anurag Gulati]

If you type "hadoop fs -ls /" it will show you the folders that currently exist 
on your hadoop cluster.

Regards,

[02AXP_4C_grad]

Anurag Gulati   |   Lead Programmer Analyst  |  Disruptive Innovation - 
Socializing Acquisition
' (602) 537-7265 | * anurag.gul...@aexp.com 
 | * 18850 N 56th St, Phoenix, AZ 85054 

%Motto: Never neglect an opportunity for improvement.   
   [cid:image002.png@01CD0C10.71371DE0] 
  [cid:image003.png@01CD0C10.71371DE0] 
  
[cid:image004.png@01CD0C10.71371DE0]   
[cid:image005.png@01CD0C10.71371DE0] 
  
[cid:image006.png@01CD0C10.71371DE0] 
  
[cid:image007.png@01CD0C10.71371DE0] 



From: Babak Bastan [mailto:babak...@gmail.com]
Sent: Tuesday, June 05, 2012 8:57 AM
To: user@hive.apache.org; Bejoy Ks
Subject: Re: Twitter Data analyse with HIVE

Thank you for your answer
location of file in lfs
That means the location of my *txt file on my computer ? and I have no 
destination address in hdfs,where can I get this location?
could you please write an example?
On Tue, Jun 5, 2012 at 4:29 PM, Bejoy Ks 
mailto:bejoy...@yahoo.com>> wrote:
Hi Babak

There isn't anything called hdfs files. Hdfs is just a file system that can 
store any type of file. You just need to transfer your file from lfs to hdfs 
and the following command helps you out for that

hadoop fs -copyFromLocal  

Regards
Bejoy KS


From: Babak Bastan mailto:babak...@gmail.com>>
To: user@hive.apache.org
Sent: Tuesday, June 5, 2012 7:54 PM
Subject: Re: Twitter Data analyse with HIVE

ok, no difference for me records in a line or not
 2009-06-08 21:49:37 - http://twitter.com/evionblablabla- I think data mining 
is awesome!
2009-06-08 21:49:37 - http://twitter.com/bliblibli 
-  I don't think so. I don't like data mining


How can I do that.I think that I should change my text file to hdfs 
file,correct? how can I do this one?
Sorry I'm very new in this field :(
On Tue, Jun 5, 2012 at 4:07 PM, Edward Capriolo 
mailto:edlinuxg...@gmail.com>> wrote:
If you get output onto a single line it will be much easier for hive to process.

On Tue, Jun 5, 2012 at 5:20 AM, Babak Bastan 
mailto:babak...@gmail.com>> wrote:
> Hi experts
>
> I'm very new in Hive and Hadoop and I want to create a very simple demo to
> analyse sample twitts like this:
>
> T 2009-06-08 21:49:37
> U http://twitter.com/evion
> W I think data mining is awesome!
>
> T 2009-06-08 21:49:37
> U http://twitter.com/hyungjin
> W I don't think so. I don't like data mining
> 
> Generally is it possible to do that?
> but I don't know exactly from which point should I strat.Do you know any
> simple and clear reference to do this job? or  would you please inform me
> (not in detail) what should I do?
>
> Thank you very much for your helps
> Babak




American Express made the following annotations on Tue Jun 05 2012 09:06:15 

** 

"This message and any attachments are solely for the intended recipient and may 
contain confidential or privileged information. If you are not the intended 
recipient, any disclosure, copying, use, or distribution of the information 
included in this message and any attachments is prohibited. If you have 
received this communication in error, please notify us by reply e-mail and 
immediately and permanently delete this message and any attachments. Thank 
you." 

American Express a ajouté le commentaire suivant le Tue Jun 05 2012 09:06:15 

Ce courrier et toute pièce jointe qu'il contient sont réservés au seul 
destinataire indiqué et peuvent renfermer des renseignements confidentiels et 
privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, 
duplication, utilisation ou distribution du courrier ou de toute pièce jointe 
est interdite. Si vous avez reçu cette communication par erreur, veuillez nous 
en aviser par courrier et détruire immédiatement le courrier et les pièces 
jointes. Merci. 

** 
---

<><><><><><><>

Re: Twitter Data analyse with HIVE

[Babak Bastan]

Thank you for your answer
location of file in lfs
That means the location of my *txt file on my computer ? and I have no
destination address in hdfs,where can I get this location?
could you please write an example?

On Tue, Jun 5, 2012 at 4:29 PM, Bejoy Ks  wrote:

> Hi Babak
>
> There isn't anything called hdfs files. Hdfs is just a file system that
> can store any type of file. You just need to transfer your file from lfs to
> hdfs and the following command helps you out for that
>
> hadoop fs -copyFromLocal   in hdfs>
>
> Regards
> Bejoy KS
>
>   --
> *From:* Babak Bastan 
> *To:* user@hive.apache.org
> *Sent:* Tuesday, June 5, 2012 7:54 PM
> *Subject:* Re: Twitter Data analyse with HIVE
>
> ok, no difference for me records in a line or not
>  2009-06-08 21:49:37 - http://twitter.com/evionblablabla- I think data
> mining is awesome!
> 2009-06-08 21:49:37 - http://twitter.com/ 
> bliblibli
> -  I don’t think so. I don’t like data mining
> 
>
> How can I do that.I think that I should change my text file to hdfs
> file,correct? how can I do this one?
> Sorry I'm very new in this field :(
>
> On Tue, Jun 5, 2012 at 4:07 PM, Edward Capriolo wrote:
>
> If you get output onto a single line it will be much easier for hive to
> process.
>
> On Tue, Jun 5, 2012 at 5:20 AM, Babak Bastan  wrote:
> > Hi experts
> >
> > I'm very new in Hive and Hadoop and I want to create a very simple demo
> to
> > analyse sample twitts like this:
> >
> > T 2009-06-08 21:49:37
> > U http://twitter.com/evion
> > W I think data mining is awesome!
> >
> > T 2009-06-08 21:49:37
> > U http://twitter.com/hyungjin
> > W I don’t think so. I don’t like data mining
> > 
> > Generally is it possible to do that?
> > but I don't know exactly from which point should I strat.Do you know any
> > simple and clear reference to do this job? or  would you please inform me
> > (not in detail) what should I do?
> >
> > Thank you very much for your helps
> > Babak
>
>
>
>
>

Re: Developing Hive UDF in eclipse

[kulkarni.swar...@gmail.com]

Did you try this[1]? It had got me most of my way through the process.

[1] https://cwiki.apache.org/Hive/gettingstarted-eclipsesetup.html

On Tue, Jun 5, 2012 at 8:49 AM, Arun Prakash wrote:

> Hi Friends,
> I tried to develop udf for hive but i am getting package import error
> in eclipse.
>
> import org.apache.hadoop.hive.ql.exec.UDF;
>
>
> How to import hive package in eclipse?
>
>
> Any inputs much appreciated.
>
>
>
> Best Regards
>  Arun Prakash C.K
>
> Keep On Sharing Your Knowledge with Others
>



-- 
Swarnim

Re: getStructFieldData method on StructObjectInspector

[kulkarni.swar...@gmail.com]

Thanks Edward for your reply on this.

Would you mind giving a very small example on how a struct corresponds to a
Map? I am having hard time understanding what the K/V pairs in the map
would look like.

Thanks again.

On Tue, May 29, 2012 at 10:16 AM, Edward Capriolo wrote:

> Returning custom writables will not work. In most cases the methods
> return Object because the types can be many things that do not fall
> under a single superclass other then object. like Integer,IntWritable,
> Array, or Map. In your case, a struct corresponds to a
> Map.
>
> On Tue, May 29, 2012 at 11:08 AM, kulkarni.swar...@gmail.com
>  wrote:
> > If someone can help understand this, I would really appreciate.
> >
> > On Fri, May 25, 2012 at 3:58 PM, kulkarni.swar...@gmail.com
> >  wrote:
> >>
> >> I am trying to write a custom ObjectInspector extending the
> >> StructObjectInspector and got a little confused about the use of the
> >> getStructFieldData method on the inspector. Looking at the definition
> of the
> >> method:
> >>
> >> public Object getStructFieldData(Object data, StructField fieldRef);
> >>
> >> I understand that the use of this method is to retrieve the specific
> given
> >> field from the buffer. However, what I don't understand is what is it
> >> expected to return. I looked around the tests and related code and
> mostly
> >> stuff returned was either a LazyPrimitive or a LazyNonPrimitive, but I
> >> couldn't find anything that enforces this(specially given that the
> return
> >> type is a plain "Object")! Does this mean that I am free to return even
> my
> >> custom object as a return type of this method? If so, what is the
> guarantee
> >> that it will be interpreted correctly down the pipeline?
> >>
> >> Thanks,
> >> --
> >> Swarnim
> >
> >
> >
> >
> > --
> > Swarnim
>



-- 
Swarnim

Re: Twitter Data analyse with HIVE

[Bejoy Ks]

Hi Babak

There isn't anything called hdfs files. Hdfs is just a file system that can 
store any type of file. You just need to transfer your file from lfs to hdfs 
and the following command helps you out for that

hadoop fs -copyFromLocal  

Regards
Bejoy KS




 From: Babak Bastan 
To: user@hive.apache.org 
Sent: Tuesday, June 5, 2012 7:54 PM
Subject: Re: Twitter Data analyse with HIVE
 

ok, no difference for me records in a line or not
 2009-06-08 21:49:37 - http://twitter.com/evionblablabla- I think data mining 
is awesome!
2009-06-08 21:49:37 - http://twitter.com/bliblibli -  I don’t think so. I don’t 
like data mining


How can I do that.I think that I should change my text file to hdfs 
file,correct? how can I do this one?
Sorry I'm very new in this field :(


On Tue, Jun 5, 2012 at 4:07 PM, Edward Capriolo  wrote:

If you get output onto a single line it will be much easier for hive to process.
>
>
>On Tue, Jun 5, 2012 at 5:20 AM, Babak Bastan  wrote:
>> Hi experts
>>
>> I'm very new in Hive and Hadoop and I want to create a very simple demo to
>> analyse sample twitts like this:
>>
>> T 2009-06-08 21:49:37
>> U http://twitter.com/evion
>> W I think data mining is awesome!
>>
>> T 2009-06-08 21:49:37
>> U http://twitter.com/hyungjin
>> W I don’t think so. I don’t like data mining
>> 
>> Generally is it possible to do that?
>> but I don't know exactly from which point should I strat.Do you know any
>> simple and clear reference to do this job? or  would you please inform me
>> (not in detail) what should I do?
>>
>> Thank you very much for your helps
>> Babak
>

Re: Twitter Data analyse with HIVE

[Babak Bastan]

ok, no difference for me records in a line or not
 2009-06-08 21:49:37 - http://twitter.com/
blablabla- I
think data mining is awesome!
2009-06-08 21:49:37 - http://twitter.com/
bliblibli
-  I don’t think so. I don’t like data mining


How can I do that.I think that I should change my text file to hdfs
file,correct? how can I do this one?
Sorry I'm very new in this field :(

On Tue, Jun 5, 2012 at 4:07 PM, Edward Capriolo wrote:

> If you get output onto a single line it will be much easier for hive to
> process.
>
> On Tue, Jun 5, 2012 at 5:20 AM, Babak Bastan  wrote:
> > Hi experts
> >
> > I'm very new in Hive and Hadoop and I want to create a very simple demo
> to
> > analyse sample twitts like this:
> >
> > T 2009-06-08 21:49:37
> > U http://twitter.com/evion
> > W I think data mining is awesome!
> >
> > T 2009-06-08 21:49:37
> > U http://twitter.com/hyungjin
> > W I don’t think so. I don’t like data mining
> > 
> > Generally is it possible to do that?
> > but I don't know exactly from which point should I strat.Do you know any
> > simple and clear reference to do this job? or  would you please inform me
> > (not in detail) what should I do?
> >
> > Thank you very much for your helps
> > Babak
>

Re: Twitter Data analyse with HIVE

[Edward Capriolo]

If you get output onto a single line it will be much easier for hive to process.

On Tue, Jun 5, 2012 at 5:20 AM, Babak Bastan  wrote:
> Hi experts
>
> I'm very new in Hive and Hadoop and I want to create a very simple demo to
> analyse sample twitts like this:
>
> T 2009-06-08 21:49:37
> U http://twitter.com/evion
> W I think data mining is awesome!
>
> T 2009-06-08 21:49:37
> U http://twitter.com/hyungjin
> W I don’t think so. I don’t like data mining
> 
> Generally is it possible to do that?
> but I don't know exactly from which point should I strat.Do you know any
> simple and clear reference to do this job? or  would you please inform me
> (not in detail) what should I do?
>
> Thank you very much for your helps
> Babak

Re: Multi-GroupBy-Insert optimization

[Jan Dolinár]

Hi Shan,

If you happen to have a lot of repeated data (in the most general
grouping), you might get some speedup by little pre-aggregation. The
following code should produce the same results as the example in your
first post:

>From (
  SELECT a, b , c, count(*) AS cnt
  FROM X
  group by a,b,c
 ) t
INSERT OVERWRITE LOCAL DIRECTORY 'output/y1'
SELECT a, b , c, cnt
INSERT OVERWRITE LOCAL DIRECTORY 'output/y2'
SELECT  a ,  SUM(cnt)
group by a
INSERT OVERWRITE LOCAL DIRECTORY 'output/y3'
SELECT b,  SUM(cnt)
group by b
...

The trick is that there there will be one more job that will first
reduce the number of records that are used in the following jobs. They
will only have to read one line for each distinct triplet a,b,c. Note
that this will only help if the number of distinct combinations is
relatively low compared to the total amount of data. In other cases it
might make no difference or even make the calculation longer.

Hope that helps... I can't think about anything else that could help you.

Jan

On 6/5/12, Jan Dolinár  wrote:
> On 6/4/12, shan s  wrote:
>> Thanks for the explanation Jan.
>> If I understand correctly, the input will be read one single time and
>> will
>> be preprocessed in some form,  and this intermediate data is used for
>> subsequent group-by..
>> Not sure if my scenario will help this single step, since group-by varies
>> across vast entities.
>
> Yes, that is that is correct. The very simplest use case is when you
> only scan a part of table. But if you are interested in all the data,
> it is not going to help you much.
>
>> If I were to implement group-by,manually, generally  we could club them
>> together in single program. Can I do better with hive, with some
>> hints/optimizations?
>> Or  is there a possibility that Pig might perform better in this case.(
>> Assuming Pig would probably handle this in a single job?)
>
> In some cases it might be able to outsmart the hive optimizer and
> write the mapreduce job directly in java in such way that it might
> perform better. In most cases though, it is probably not worth the
> trouble. You might easily end up in situation where buying more
> machines is cheaper than developing the low level solutions that might
> or might not be slightly faster... I'm not familiar with Pig or any
> other tools that might be of use in your situation.
>
> Jan
>

Twitter Data analyse with HIVE

[Babak Bastan]

Hi experts

I'm very new in Hive and Hadoop and I want to create a very simple demo to
analyse sample twitts like this:

T 2009-06-08 21:49:37
U http://twitter.com/evion
W I think data mining is awesome!

T 2009-06-08 21:49:37
U http://twitter.com/hyungjin
W I don’t think so. I don’t like data mining

Generally is it possible to do that?
but I don't know exactly from which point should I strat.Do you know any
simple and clear reference to do this job? or  would you please inform me
(not in detail) what should I do?

Thank you very much for your helps
Babak