date:20120606

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

follow this link -
http://hadoop.apache.org/common/docs/r0.20.203.0/single_node_setup.html..it
worked for most of us without any prooblem.

do all the things required to configure hadoop on linux in pseudo
distributed mode as given in this link..start with a simple setup as
shown there..then we'll add more properties

if you need detailed help you can also visit -
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/

Regards,
    Mohammad Tariq


On Thu, Jun 7, 2012 at 3:04 AM, Babak Bastan  wrote:
> I try to install another one in blog,most of them was performed without
> problem but in this step
>
>  udo mkdir /hadoop && sudo chown hdfs:hdfs /hadoop && sudo chmod 777 /hadoop
>
> I get this error:
>
> Error:  0: couldn't open source file 
>
> and in this step:
> mkdir /usr/lib/hadoop-0.20/.ssh
> this error:
> mkdir: kann Verzeichnis „/usr/lib/hadoop-0.20/.ssh“ nicht anlegen: Keine
> Berechtigung
> ---> no permission to make a directory
>
> On Wed, Jun 6, 2012 at 11:21 PM, Mohammad Tariq  wrote:
>>
>> ok..we'll give it a final shot..then i'll email configured hadoop to
>> your email address..delete the hdfs directory which contains tmp, data
>> and name..recreate it..format hdfs again and then start the processes.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Jun 7, 2012 at 2:22 AM, Babak Bastan  wrote:
>> > I 've performed the steps but the same error in this step as before:
>> > bin/start-dfs.sh
>> > It is about my permission to make directory
>> >
>> > On Wed, Jun 6, 2012 at 10:33 PM, Mohammad Tariq 
>> > wrote:
>> >>
>> >> actually this blog post explains how to install cloudera's hadoop
>> >> distribution...if you have followed this post and installed cloudera's
>> >> distribution then your logs should ideally be inside
>> >> /usr/lib/hadoop/logs (if everything was fine)..anyway try the steps I
>> >> have given and let me know.
>> >>
>> >> Regards,
>> >>     Mohammad Tariq
>> >>
>> >>
>> >> On Thu, Jun 7, 2012 at 1:52 AM, Babak Bastan 
>> >> wrote:
>> >> > by the way,you are a very nice man my friend:Thank you so much :)
>> >> >
>> >> > what do you mean aboat this post in stackoverflow?
>> >> >
>> >> > I am assuming that is your first installation of hadoop.
>> >> >
>> >> > At the beginning please check if your daemons are working. To do that
>> >> > use
>> >> > (in terminal):
>> >> >
>> >> > jps
>> >> >
>> >> > If only jps appears that means all daemons are down. Please check the
>> >> > log
>> >> > files. Especially the namenode. Log folder is probably somewhere
>> >> > there
>> >> > /usr/lib/hadoop/logs
>> >> >
>> >> > If you have some permission problems. Use this guide during the
>> >> > installation.
>> >> >
>> >> > Good installation guide
>> >> >
>> >> > I am shooting with this explanations but these are most common
>> >> > problems.
>> >> >
>> >> >
>> >> > On Wed, Jun 6, 2012 at 10:15 PM, Babak Bastan 
>> >> > wrote:
>> >> >>
>> >> >> I checked it but no hadoop folder :(
>> >> >> yes you are right.I'm a student and I want to make a very very
>> >> >> simple
>> >> >> programm hive but untill now hm
>> >> >>
>> >> >>
>> >> >> On Wed, Jun 6, 2012 at 10:12 PM, Babak Bastan 
>> >> >> wrote:
>> >> >>>
>> >> >>> no one error:
>> >> >>> i.e if I run this one
>> >> >>>
>> >> >>> hostname --fqdn
>> >> >>>
>> >> >>>  with the condition that I send to you :
>> >> >>>
>> >> >>> 127.0.0.1       localhost
>> >> >>> #127.0.0.1      ubuntu.ubuntu-domain    ubuntu
>> >> >>> # The following lines are desirable for IPv6 capable hosts
>> >> >>> #::1     ip6-localhost ip6-loopback
>> >> >>> #fe00::0 ip6-localnet
>> >> >>> #ff00::0 ip6-mcastprefix
>> >> >>> #ff02::1 ip6-allnodes
>> >> >>> #ff02::2 ip6-allrouters
>> >> >>>
>> >> >>> I get this error:
>> >> >>>
>> >> >>> hostname: Name or service not known
>> >> >>>
>> >> >>> Or in the second step by this command:
>> >> >>>
>> >> >>> babak@ubuntu:~/Downloads/hadoop/bin$ start-hdfs.sh
>> >> >>>
>> >> >>> these lines of error:
>> >> >>>
>> >> >>>
>> >> >>> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“
>> >> >>> nicht
>> >> >>> anlegen: Keine Berechtigung
>> >> >>> starting namenode, logging to
>> >> >>>
>> >> >>>
>> >> >>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out
>> >> >>> /home/babak/Downloads/hadoop/bin/hadoop-daemon.sh: Zeile 117:
>> >> >>>
>> >> >>>
>> >> >>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out:
>> >> >>> Datei oder Verzeichnis nicht gefunden
>> >> >>> head:
>> >> >>>
>> >> >>>
>> >> >>> „/home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out“
>> >> >>> kann nicht zum Lesen geöffnet werden: Datei oder Verzeichnis nicht
>> >> >>> gefunden
>> >> >>> localhost: mkdir: kann Verzeichnis
>> >> >>> „/home/babak/Downloads/hadoop/bin/../logs“ nicht anlegen: Keine
>> >> >>> Berechtigung
>> >> >>> localhost: starting datanode, logging to
>> >> >>>
>> >> >>>
>> >> >>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-b

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

ok..we'll give it a final shot..then i'll email configured hadoop to
your email address..delete the hdfs directory which contains tmp, data
and name..recreate it..format hdfs again and then start the processes.

Regards,
    Mohammad Tariq


On Thu, Jun 7, 2012 at 2:22 AM, Babak Bastan  wrote:
> I 've performed the steps but the same error in this step as before:
> bin/start-dfs.sh
> It is about my permission to make directory
>
> On Wed, Jun 6, 2012 at 10:33 PM, Mohammad Tariq  wrote:
>>
>> actually this blog post explains how to install cloudera's hadoop
>> distribution...if you have followed this post and installed cloudera's
>> distribution then your logs should ideally be inside
>> /usr/lib/hadoop/logs (if everything was fine)..anyway try the steps I
>> have given and let me know.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Jun 7, 2012 at 1:52 AM, Babak Bastan  wrote:
>> > by the way,you are a very nice man my friend:Thank you so much :)
>> >
>> > what do you mean aboat this post in stackoverflow?
>> >
>> > I am assuming that is your first installation of hadoop.
>> >
>> > At the beginning please check if your daemons are working. To do that
>> > use
>> > (in terminal):
>> >
>> > jps
>> >
>> > If only jps appears that means all daemons are down. Please check the
>> > log
>> > files. Especially the namenode. Log folder is probably somewhere there
>> > /usr/lib/hadoop/logs
>> >
>> > If you have some permission problems. Use this guide during the
>> > installation.
>> >
>> > Good installation guide
>> >
>> > I am shooting with this explanations but these are most common problems.
>> >
>> >
>> > On Wed, Jun 6, 2012 at 10:15 PM, Babak Bastan 
>> > wrote:
>> >>
>> >> I checked it but no hadoop folder :(
>> >> yes you are right.I'm a student and I want to make a very very simple
>> >> programm hive but untill now hm
>> >>
>> >>
>> >> On Wed, Jun 6, 2012 at 10:12 PM, Babak Bastan 
>> >> wrote:
>> >>>
>> >>> no one error:
>> >>> i.e if I run this one
>> >>>
>> >>> hostname --fqdn
>> >>>
>> >>>  with the condition that I send to you :
>> >>>
>> >>> 127.0.0.1       localhost
>> >>> #127.0.0.1      ubuntu.ubuntu-domain    ubuntu
>> >>> # The following lines are desirable for IPv6 capable hosts
>> >>> #::1     ip6-localhost ip6-loopback
>> >>> #fe00::0 ip6-localnet
>> >>> #ff00::0 ip6-mcastprefix
>> >>> #ff02::1 ip6-allnodes
>> >>> #ff02::2 ip6-allrouters
>> >>>
>> >>> I get this error:
>> >>>
>> >>> hostname: Name or service not known
>> >>>
>> >>> Or in the second step by this command:
>> >>>
>> >>> babak@ubuntu:~/Downloads/hadoop/bin$ start-hdfs.sh
>> >>>
>> >>> these lines of error:
>> >>>
>> >>>
>> >>> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“
>> >>> nicht
>> >>> anlegen: Keine Berechtigung
>> >>> starting namenode, logging to
>> >>>
>> >>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out
>> >>> /home/babak/Downloads/hadoop/bin/hadoop-daemon.sh: Zeile 117:
>> >>>
>> >>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out:
>> >>> Datei oder Verzeichnis nicht gefunden
>> >>> head:
>> >>>
>> >>> „/home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out“
>> >>> kann nicht zum Lesen geöffnet werden: Datei oder Verzeichnis nicht
>> >>> gefunden
>> >>> localhost: mkdir: kann Verzeichnis
>> >>> „/home/babak/Downloads/hadoop/bin/../logs“ nicht anlegen: Keine
>> >>> Berechtigung
>> >>> localhost: starting datanode, logging to
>> >>>
>> >>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-datanode-ubuntu.out
>> >>> localhost: /home/babak/Downloads/hadoop/bin/hadoop-daemon.sh: Zeile
>> >>> 117:
>> >>>
>> >>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-datanode-ubuntu.out:
>> >>> Datei oder Verzeichnis nicht gefunden
>> >>> localhost: head:
>> >>>
>> >>> „/home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-datanode-ubuntu.out“
>> >>> kann nicht zum Lesen geöffnet werden: Datei oder Verzeichnis nicht
>> >>> gefunden
>> >>> localhost: mkdir: kann Verzeichnis
>> >>> „/home/babak/Downloads/hadoop/bin/../logs“ nicht anlegen: Keine
>> >>> Berechtigung
>> >>> localhost: starting secondarynamenode, logging to
>> >>>
>> >>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-secondarynamenode-ubuntu.out
>> >>> localhost: /home/babak/Downloads/hadoop/bin/hadoop-daemon.sh: Zeile
>> >>> 117:
>> >>>
>> >>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-secondarynamenode-ubuntu.out:
>> >>> Datei oder Verzeichnis nicht gefunden
>> >>> localhost: head:
>> >>>
>> >>> „/home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-secondarynamenode-ubuntu.out“
>> >>> kann nicht zum Lesen geöffnet werden: Datei oder Verzeichnis nicht
>> >>> gefunden
>> >>>
>> >>> they said no permision to make logs in this
>> >>> path:/home/babak/Downloads/hadoop/bin/../logs
>> >>>
>> >>>  and generally I cant create a table in hive and get this one:
>> >>>
>> >>> FAILED: Error in metadata: MetaException(message:Got exception:
>> >>> j

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

actually this blog post explains how to install cloudera's hadoop
distribution...if you have followed this post and installed cloudera's
distribution then your logs should ideally be inside
/usr/lib/hadoop/logs (if everything was fine)..anyway try the steps I
have given and let me know.

Regards,
    Mohammad Tariq


On Thu, Jun 7, 2012 at 1:52 AM, Babak Bastan  wrote:
> by the way,you are a very nice man my friend:Thank you so much :)
>
> what do you mean aboat this post in stackoverflow?
>
> I am assuming that is your first installation of hadoop.
>
> At the beginning please check if your daemons are working. To do that use
> (in terminal):
>
> jps
>
> If only jps appears that means all daemons are down. Please check the log
> files. Especially the namenode. Log folder is probably somewhere there
> /usr/lib/hadoop/logs
>
> If you have some permission problems. Use this guide during the
> installation.
>
> Good installation guide
>
> I am shooting with this explanations but these are most common problems.
>
>
> On Wed, Jun 6, 2012 at 10:15 PM, Babak Bastan  wrote:
>>
>> I checked it but no hadoop folder :(
>> yes you are right.I'm a student and I want to make a very very simple
>> programm hive but untill now hm
>>
>>
>> On Wed, Jun 6, 2012 at 10:12 PM, Babak Bastan  wrote:
>>>
>>> no one error:
>>> i.e if I run this one
>>>
>>> hostname --fqdn
>>>
>>>  with the condition that I send to you :
>>>
>>> 127.0.0.1       localhost
>>> #127.0.0.1      ubuntu.ubuntu-domain    ubuntu
>>> # The following lines are desirable for IPv6 capable hosts
>>> #::1     ip6-localhost ip6-loopback
>>> #fe00::0 ip6-localnet
>>> #ff00::0 ip6-mcastprefix
>>> #ff02::1 ip6-allnodes
>>> #ff02::2 ip6-allrouters
>>>
>>> I get this error:
>>>
>>> hostname: Name or service not known
>>>
>>> Or in the second step by this command:
>>>
>>> babak@ubuntu:~/Downloads/hadoop/bin$ start-hdfs.sh
>>>
>>> these lines of error:
>>>
>>>
>>> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“ nicht
>>> anlegen: Keine Berechtigung
>>> starting namenode, logging to
>>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out
>>> /home/babak/Downloads/hadoop/bin/hadoop-daemon.sh: Zeile 117:
>>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out:
>>> Datei oder Verzeichnis nicht gefunden
>>> head:
>>> „/home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out“
>>> kann nicht zum Lesen geöffnet werden: Datei oder Verzeichnis nicht gefunden
>>> localhost: mkdir: kann Verzeichnis
>>> „/home/babak/Downloads/hadoop/bin/../logs“ nicht anlegen: Keine Berechtigung
>>> localhost: starting datanode, logging to
>>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-datanode-ubuntu.out
>>> localhost: /home/babak/Downloads/hadoop/bin/hadoop-daemon.sh: Zeile 117:
>>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-datanode-ubuntu.out:
>>> Datei oder Verzeichnis nicht gefunden
>>> localhost: head:
>>> „/home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-datanode-ubuntu.out“
>>> kann nicht zum Lesen geöffnet werden: Datei oder Verzeichnis nicht gefunden
>>> localhost: mkdir: kann Verzeichnis
>>> „/home/babak/Downloads/hadoop/bin/../logs“ nicht anlegen: Keine Berechtigung
>>> localhost: starting secondarynamenode, logging to
>>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-secondarynamenode-ubuntu.out
>>> localhost: /home/babak/Downloads/hadoop/bin/hadoop-daemon.sh: Zeile 117:
>>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-secondarynamenode-ubuntu.out:
>>> Datei oder Verzeichnis nicht gefunden
>>> localhost: head:
>>> „/home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-secondarynamenode-ubuntu.out“
>>> kann nicht zum Lesen geöffnet werden: Datei oder Verzeichnis nicht gefunden
>>>
>>> they said no permision to make logs in this
>>> path:/home/babak/Downloads/hadoop/bin/../logs
>>>
>>>  and generally I cant create a table in hive and get this one:
>>>
>>> FAILED: Error in metadata: MetaException(message:Got exception:
>>> java.io.FileNotFoundException File file:/user/hive/warehouse/test does not
>>> exist.)
>>> FAILED: Execution Error, return code 1 from
>>> org.apache.hadoop.hive.ql.exec.DDLTask
>>>
>>> On Wed, Jun 6, 2012 at 10:02 PM, shashwat shriparv
>>>  wrote:

 whats the error babak ???


 On Thu, Jun 7, 2012 at 1:25 AM, Babak Bastan  wrote:
>
> What the hell is that?I see no log folder there
>
>
> On Wed, Jun 6, 2012 at 9:41 PM, Mohammad Tariq 
> wrote:
>>
>> go to your HADOOP_HOME i.e your hadoop directory(that includes bin,
>> conf etc)..you can find logs directory there..
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Jun 7, 2012 at 1:09 AM, Babak Bastan 
>> wrote:
>> > hoe can I get my log mohammad?
>> >
>> >
>> > On Wed, Jun 6, 2012 at 9:36 PM, Mohammad Tariq 
>> > wrote:
>> >>
>> >> could you post your logs???that would help me i

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

need not worry.. i am also a student..just keep your calm..start fresh
and follow these steps -

1 - download hadoop from apache using this link -
http://apache.techartifact.com/mirror/hadoop/common/hadoop-0.20.205.0/hadoop-0.20.205.0.tar.gz

2 - untar it - right click+extract here

3 - set JAVA_HOME in your hadoop-env.sh file and save it

4 - add the properties specified in previous replies in your
core-site.xml, hdfs-site.xml and mapredif it doesn't work still i'll
send you the configured hadoop-site.xml files

5 - format HDFS

6 - start the hadoop processes

also your hosts file should look like this -

127.0.0.1   localhost
127.0.0.1   ubuntu.ubuntu-domainubuntu

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

it'll work..if you further face any problem, i'll send you a
configured copy of hadoop.

Regards,
    Mohammad Tariq


On Thu, Jun 7, 2012 at 1:45 AM, Babak Bastan  wrote:
> I checked it but no hadoop folder :(
> yes you are right.I'm a student and I want to make a very very simple
> programm hive but untill now hm
>
>
> On Wed, Jun 6, 2012 at 10:12 PM, Babak Bastan  wrote:
>>
>> no one error:
>> i.e if I run this one
>>
>> hostname --fqdn
>>
>>  with the condition that I send to you :
>>
>> 127.0.0.1       localhost
>> #127.0.0.1      ubuntu.ubuntu-domain    ubuntu
>> # The following lines are desirable for IPv6 capable hosts
>> #::1     ip6-localhost ip6-loopback
>> #fe00::0 ip6-localnet
>> #ff00::0 ip6-mcastprefix
>> #ff02::1 ip6-allnodes
>> #ff02::2 ip6-allrouters
>>
>> I get this error:
>>
>> hostname: Name or service not known
>>
>> Or in the second step by this command:
>>
>> babak@ubuntu:~/Downloads/hadoop/bin$ start-hdfs.sh
>>
>> these lines of error:
>>
>>
>> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“ nicht
>> anlegen: Keine Berechtigung
>> starting namenode, logging to
>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out
>> /home/babak/Downloads/hadoop/bin/hadoop-daemon.sh: Zeile 117:
>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out:
>> Datei oder Verzeichnis nicht gefunden
>> head:
>> „/home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-namenode-ubuntu.out“
>> kann nicht zum Lesen geöffnet werden: Datei oder Verzeichnis nicht gefunden
>> localhost: mkdir: kann Verzeichnis
>> „/home/babak/Downloads/hadoop/bin/../logs“ nicht anlegen: Keine Berechtigung
>> localhost: starting datanode, logging to
>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-datanode-ubuntu.out
>> localhost: /home/babak/Downloads/hadoop/bin/hadoop-daemon.sh: Zeile 117:
>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-datanode-ubuntu.out:
>> Datei oder Verzeichnis nicht gefunden
>> localhost: head:
>> „/home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-datanode-ubuntu.out“
>> kann nicht zum Lesen geöffnet werden: Datei oder Verzeichnis nicht gefunden
>> localhost: mkdir: kann Verzeichnis
>> „/home/babak/Downloads/hadoop/bin/../logs“ nicht anlegen: Keine Berechtigung
>> localhost: starting secondarynamenode, logging to
>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-secondarynamenode-ubuntu.out
>> localhost: /home/babak/Downloads/hadoop/bin/hadoop-daemon.sh: Zeile 117:
>> /home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-secondarynamenode-ubuntu.out:
>> Datei oder Verzeichnis nicht gefunden
>> localhost: head:
>> „/home/babak/Downloads/hadoop/bin/../logs/hadoop-babak-secondarynamenode-ubuntu.out“
>> kann nicht zum Lesen geöffnet werden: Datei oder Verzeichnis nicht gefunden
>>
>> they said no permision to make logs in this
>> path:/home/babak/Downloads/hadoop/bin/../logs
>>
>>  and generally I cant create a table in hive and get this one:
>>
>> FAILED: Error in metadata: MetaException(message:Got exception:
>> java.io.FileNotFoundException File file:/user/hive/warehouse/test does not
>> exist.)
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.DDLTask
>>
>> On Wed, Jun 6, 2012 at 10:02 PM, shashwat shriparv
>>  wrote:
>>>
>>> whats the error babak ???
>>>
>>>
>>> On Thu, Jun 7, 2012 at 1:25 AM, Babak Bastan  wrote:

 What the hell is that?I see no log folder there


 On Wed, Jun 6, 2012 at 9:41 PM, Mohammad Tariq 
 wrote:
>
> go to your HADOOP_HOME i.e your hadoop directory(that includes bin,
> conf etc)..you can find logs directory there..
>
> Regards,
>     Mohammad Tariq
>
>
> On Thu, Jun 7, 2012 at 1:09 AM, Babak Bastan 
> wrote:
> > hoe can I get my log mohammad?
> >
> >
> > On Wed, Jun 6, 2012 at 9:36 PM, Mohammad Tariq 
> > wrote:
> >>
> >> could you post your logs???that would help me in understanding the
> >> problem properly.
> >>
> >> Regards,
> >>     Mohammad Tariq
> >>
> >>
> >> On Thu, Jun 7, 201

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

check your /var/log/hadoop/...also when you do something wrong your
will find your terminal full of many error messages, you can use them
as well..and by the way learning something new requires great deal of
patience

Regards,
    Mohammad Tariq


On Thu, Jun 7, 2012 at 1:25 AM, Babak Bastan  wrote:
> What the hell is that?I see no log folder there
>
>
> On Wed, Jun 6, 2012 at 9:41 PM, Mohammad Tariq  wrote:
>>
>> go to your HADOOP_HOME i.e your hadoop directory(that includes bin,
>> conf etc)..you can find logs directory there..
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Jun 7, 2012 at 1:09 AM, Babak Bastan  wrote:
>> > hoe can I get my log mohammad?
>> >
>> >
>> > On Wed, Jun 6, 2012 at 9:36 PM, Mohammad Tariq 
>> > wrote:
>> >>
>> >> could you post your logs???that would help me in understanding the
>> >> problem properly.
>> >>
>> >> Regards,
>> >>     Mohammad Tariq
>> >>
>> >>
>> >> On Thu, Jun 7, 2012 at 1:02 AM, Babak Bastan 
>> >> wrote:
>> >> > Thank you very much mohamad for your attention.I followed the steps
>> >> > but
>> >> > the
>> >> > error is the same as the last time.
>> >> > and there is my hosts file:
>> >> >
>> >> > 127.0.0.1       localhost
>> >> > #127.0.0.1      ubuntu.ubuntu-domain    ubuntu
>> >> >
>> >> >
>> >> > # The following lines are desirable for IPv6 capable hosts
>> >> >
>> >> > #::1     ip6-localhost ip6-loopback
>> >> > #fe00::0 ip6-localnet
>> >> > #ff00::0 ip6-mcastprefix
>> >> > #ff02::1 ip6-allnodes
>> >> > #ff02::2 ip6-allrouters
>> >> >
>> >> > but no effect :(
>> >> >
>> >> > On Wed, Jun 6, 2012 at 8:25 PM, Mohammad Tariq 
>> >> > wrote:
>> >> >>
>> >> >> also change the permissions of these directories to 777.
>> >> >>
>> >> >> Regards,
>> >> >>     Mohammad Tariq
>> >> >>
>> >> >>
>> >> >> On Wed, Jun 6, 2012 at 11:54 PM, Mohammad Tariq 
>> >> >> wrote:
>> >> >> > create a directory "/home/username/hdfs" (or at some place of your
>> >> >> > choice)..inside this hdfs directory create three sub directories -
>> >> >> > name, data, and temp, then follow these steps :
>> >> >> >
>> >> >> > add following properties in your core-site.xml -
>> >> >> >
>> >> >> > 
>> >> >> >          fs.default.name
>> >> >> >          hdfs://localhost:9000/
>> >> >> >        
>> >> >> >
>> >> >> >        
>> >> >> >          hadoop.tmp.dir
>> >> >> >          /home/mohammad/hdfs/temp
>> >> >> >        
>> >> >> >
>> >> >> > then add following two properties in your hdfs-site.xml -
>> >> >> >
>> >> >> > 
>> >> >> >                dfs.replication
>> >> >> >                1
>> >> >> >        
>> >> >> >
>> >> >> >        
>> >> >> >                dfs.name.dir
>> >> >> >                /home/mohammad/hdfs/name
>> >> >> >        
>> >> >> >
>> >> >> >        
>> >> >> >                dfs.data.dir
>> >> >> >                /home/mohammad/hdfs/data
>> >> >> >        
>> >> >> >
>> >> >> > finally add this property in your mapred-site.xml -
>> >> >> >
>> >> >> >       
>> >> >> >          mapred.job.tracker
>> >> >> >          hdfs://localhost:9001
>> >> >> >        
>> >> >> >
>> >> >> > NOTE: you can give any name to these directories of your choice,
>> >> >> > just
>> >> >> > keep in mind you have to give same names as values of
>> >> >> >           above specified properties in your configuration files.
>> >> >> > (give full path of these directories, not just the name of the
>> >> >> > directory)
>> >> >> >
>> >> >> > After this  follow the steps provided in the previous reply.
>> >> >> >
>> >> >> > Regards,
>> >> >> >     Mohammad Tariq
>> >> >> >
>> >> >> >
>> >> >> > On Wed, Jun 6, 2012 at 11:42 PM, Babak Bastan 
>> >> >> > wrote:
>> >> >> >> thank's Mohammad
>> >> >> >>
>> >> >> >> with this command:
>> >> >> >>
>> >> >> >> babak@ubuntu:~/Downloads/hadoop/bin$ hadoop namenode -format
>> >> >> >>
>> >> >> >> this is my output:
>> >> >> >>
>> >> >> >> 12/06/06 20:05:20 INFO namenode.NameNode: STARTUP_MSG:
>> >> >> >> /
>> >> >> >> STARTUP_MSG: Starting NameNode
>> >> >> >> STARTUP_MSG:   host = ubuntu/127.0.1.1
>> >> >> >> STARTUP_MSG:   args = [-format]
>> >> >> >> STARTUP_MSG:   version = 0.20.2
>> >> >> >> STARTUP_MSG:   build =
>> >> >> >>
>> >> >> >> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>> >> >> >> -r
>> >> >> >> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
>> >> >> >> /
>> >> >> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
>> >> >> >>
>> >> >> >>
>> >> >> >> fsOwner=babak,babak,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
>> >> >> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
>> >> >> >> supergroup=supergroup
>> >> >> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
>> >> >> >> isPermissionEnabled=true
>> >> >> >> 12/06/06 20:05:20 INFO common.Storage: Image file of size 95
>> >> >> >> saved
>> >> >> >> in 0
>> >> >> >> seconds.
>> >> >> >> 12/06/06 20:05:20 INFO common.Storage: Storage

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

go to your HADOOP_HOME i.e your hadoop directory(that includes bin,
conf etc)..you can find logs directory there..

Regards,
    Mohammad Tariq


On Thu, Jun 7, 2012 at 1:09 AM, Babak Bastan  wrote:
> hoe can I get my log mohammad?
>
>
> On Wed, Jun 6, 2012 at 9:36 PM, Mohammad Tariq  wrote:
>>
>> could you post your logs???that would help me in understanding the
>> problem properly.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Thu, Jun 7, 2012 at 1:02 AM, Babak Bastan  wrote:
>> > Thank you very much mohamad for your attention.I followed the steps but
>> > the
>> > error is the same as the last time.
>> > and there is my hosts file:
>> >
>> > 127.0.0.1       localhost
>> > #127.0.0.1      ubuntu.ubuntu-domain    ubuntu
>> >
>> >
>> > # The following lines are desirable for IPv6 capable hosts
>> >
>> > #::1     ip6-localhost ip6-loopback
>> > #fe00::0 ip6-localnet
>> > #ff00::0 ip6-mcastprefix
>> > #ff02::1 ip6-allnodes
>> > #ff02::2 ip6-allrouters
>> >
>> > but no effect :(
>> >
>> > On Wed, Jun 6, 2012 at 8:25 PM, Mohammad Tariq 
>> > wrote:
>> >>
>> >> also change the permissions of these directories to 777.
>> >>
>> >> Regards,
>> >>     Mohammad Tariq
>> >>
>> >>
>> >> On Wed, Jun 6, 2012 at 11:54 PM, Mohammad Tariq 
>> >> wrote:
>> >> > create a directory "/home/username/hdfs" (or at some place of your
>> >> > choice)..inside this hdfs directory create three sub directories -
>> >> > name, data, and temp, then follow these steps :
>> >> >
>> >> > add following properties in your core-site.xml -
>> >> >
>> >> > 
>> >> >          fs.default.name
>> >> >          hdfs://localhost:9000/
>> >> >        
>> >> >
>> >> >        
>> >> >          hadoop.tmp.dir
>> >> >          /home/mohammad/hdfs/temp
>> >> >        
>> >> >
>> >> > then add following two properties in your hdfs-site.xml -
>> >> >
>> >> > 
>> >> >                dfs.replication
>> >> >                1
>> >> >        
>> >> >
>> >> >        
>> >> >                dfs.name.dir
>> >> >                /home/mohammad/hdfs/name
>> >> >        
>> >> >
>> >> >        
>> >> >                dfs.data.dir
>> >> >                /home/mohammad/hdfs/data
>> >> >        
>> >> >
>> >> > finally add this property in your mapred-site.xml -
>> >> >
>> >> >       
>> >> >          mapred.job.tracker
>> >> >          hdfs://localhost:9001
>> >> >        
>> >> >
>> >> > NOTE: you can give any name to these directories of your choice, just
>> >> > keep in mind you have to give same names as values of
>> >> >           above specified properties in your configuration files.
>> >> > (give full path of these directories, not just the name of the
>> >> > directory)
>> >> >
>> >> > After this  follow the steps provided in the previous reply.
>> >> >
>> >> > Regards,
>> >> >     Mohammad Tariq
>> >> >
>> >> >
>> >> > On Wed, Jun 6, 2012 at 11:42 PM, Babak Bastan 
>> >> > wrote:
>> >> >> thank's Mohammad
>> >> >>
>> >> >> with this command:
>> >> >>
>> >> >> babak@ubuntu:~/Downloads/hadoop/bin$ hadoop namenode -format
>> >> >>
>> >> >> this is my output:
>> >> >>
>> >> >> 12/06/06 20:05:20 INFO namenode.NameNode: STARTUP_MSG:
>> >> >> /
>> >> >> STARTUP_MSG: Starting NameNode
>> >> >> STARTUP_MSG:   host = ubuntu/127.0.1.1
>> >> >> STARTUP_MSG:   args = [-format]
>> >> >> STARTUP_MSG:   version = 0.20.2
>> >> >> STARTUP_MSG:   build =
>> >> >> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
>> >> >> -r
>> >> >> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
>> >> >> /
>> >> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
>> >> >>
>> >> >> fsOwner=babak,babak,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
>> >> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem: supergroup=supergroup
>> >> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
>> >> >> isPermissionEnabled=true
>> >> >> 12/06/06 20:05:20 INFO common.Storage: Image file of size 95 saved
>> >> >> in 0
>> >> >> seconds.
>> >> >> 12/06/06 20:05:20 INFO common.Storage: Storage directory
>> >> >> /tmp/hadoop-babak/dfs/name has been successfully formatted.
>> >> >> 12/06/06 20:05:20 INFO namenode.NameNode: SHUTDOWN_MSG:
>> >> >> /
>> >> >> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
>> >> >> /
>> >> >>
>> >> >> by this command:
>> >> >>
>> >> >> babak@ubuntu:~/Downloads/hadoop/bin$ start-dfs.sh
>> >> >>
>> >> >> this is the out put
>> >> >>
>> >> >> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“
>> >> >> nicht
>> >> >> anlegen: Keine Berechtigung
>> >> >>
>> >> >> this out put(it's in german and it means no right to make this
>> >> >> folder)
>> >> >>
>> >> >>
>> >> >> On Wed, Jun 6, 2012 at 7:59 PM, Mohammad Tariq 
>> >> >> wrote:
>> >> >>>
>> >> >>> once we are done with the configuration, we need to

Re: Error while Creating Table in Hive

2012-06-06 Thread Babak Bastan

hoe can I get my log mohammad?

On Wed, Jun 6, 2012 at 9:36 PM, Mohammad Tariq  wrote:

> could you post your logs???that would help me in understanding the
> problem properly.
>
> Regards,
> Mohammad Tariq
>
>
> On Thu, Jun 7, 2012 at 1:02 AM, Babak Bastan  wrote:
> > Thank you very much mohamad for your attention.I followed the steps but
> the
> > error is the same as the last time.
> > and there is my hosts file:
> >
> > 127.0.0.1   localhost
> > #127.0.0.1  ubuntu.ubuntu-domainubuntu
> >
> >
> > # The following lines are desirable for IPv6 capable hosts
> >
> > #::1 ip6-localhost ip6-loopback
> > #fe00::0 ip6-localnet
> > #ff00::0 ip6-mcastprefix
> > #ff02::1 ip6-allnodes
> > #ff02::2 ip6-allrouters
> >
> > but no effect :(
> >
> > On Wed, Jun 6, 2012 at 8:25 PM, Mohammad Tariq 
> wrote:
> >>
> >> also change the permissions of these directories to 777.
> >>
> >> Regards,
> >> Mohammad Tariq
> >>
> >>
> >> On Wed, Jun 6, 2012 at 11:54 PM, Mohammad Tariq 
> >> wrote:
> >> > create a directory "/home/username/hdfs" (or at some place of your
> >> > choice)..inside this hdfs directory create three sub directories -
> >> > name, data, and temp, then follow these steps :
> >> >
> >> > add following properties in your core-site.xml -
> >> >
> >> > 
> >> >  fs.default.name
> >> >  hdfs://localhost:9000/
> >> >
> >> >
> >> >
> >> >  hadoop.tmp.dir
> >> >  /home/mohammad/hdfs/temp
> >> >
> >> >
> >> > then add following two properties in your hdfs-site.xml -
> >> >
> >> > 
> >> >dfs.replication
> >> >1
> >> >
> >> >
> >> >
> >> >dfs.name.dir
> >> >/home/mohammad/hdfs/name
> >> >
> >> >
> >> >
> >> >dfs.data.dir
> >> >/home/mohammad/hdfs/data
> >> >
> >> >
> >> > finally add this property in your mapred-site.xml -
> >> >
> >> >   
> >> >  mapred.job.tracker
> >> >  hdfs://localhost:9001
> >> >
> >> >
> >> > NOTE: you can give any name to these directories of your choice, just
> >> > keep in mind you have to give same names as values of
> >> >   above specified properties in your configuration files.
> >> > (give full path of these directories, not just the name of the
> >> > directory)
> >> >
> >> > After this  follow the steps provided in the previous reply.
> >> >
> >> > Regards,
> >> > Mohammad Tariq
> >> >
> >> >
> >> > On Wed, Jun 6, 2012 at 11:42 PM, Babak Bastan 
> >> > wrote:
> >> >> thank's Mohammad
> >> >>
> >> >> with this command:
> >> >>
> >> >> babak@ubuntu:~/Downloads/hadoop/bin$ hadoop namenode -format
> >> >>
> >> >> this is my output:
> >> >>
> >> >> 12/06/06 20:05:20 INFO namenode.NameNode: STARTUP_MSG:
> >> >> /
> >> >> STARTUP_MSG: Starting NameNode
> >> >> STARTUP_MSG:   host = ubuntu/127.0.1.1
> >> >> STARTUP_MSG:   args = [-format]
> >> >> STARTUP_MSG:   version = 0.20.2
> >> >> STARTUP_MSG:   build =
> >> >> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-r
> >> >> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> >> >> /
> >> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
> >> >>
> fsOwner=babak,babak,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
> >> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem: supergroup=supergroup
> >> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
> isPermissionEnabled=true
> >> >> 12/06/06 20:05:20 INFO common.Storage: Image file of size 95 saved
> in 0
> >> >> seconds.
> >> >> 12/06/06 20:05:20 INFO common.Storage: Storage directory
> >> >> /tmp/hadoop-babak/dfs/name has been successfully formatted.
> >> >> 12/06/06 20:05:20 INFO namenode.NameNode: SHUTDOWN_MSG:
> >> >> /
> >> >> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
> >> >> /
> >> >>
> >> >> by this command:
> >> >>
> >> >> babak@ubuntu:~/Downloads/hadoop/bin$ start-dfs.sh
> >> >>
> >> >> this is the out put
> >> >>
> >> >> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“
> >> >> nicht
> >> >> anlegen: Keine Berechtigung
> >> >>
> >> >> this out put(it's in german and it means no right to make this
> folder)
> >> >>
> >> >>
> >> >> On Wed, Jun 6, 2012 at 7:59 PM, Mohammad Tariq 
> >> >> wrote:
> >> >>>
> >> >>> once we are done with the configuration, we need to format the file
> >> >>> system..use this command to do that-
> >> >>> bin/hadoop namenode -format
> >> >>>
> >> >>> after this, hadoop daemon processes should be started using
> following
> >> >>> commands -
> >> >>> bin/start-dfs.sh (it'll start NN & DN)
> >> >>> bin/start-mapred.sh (it'll start JT & TT)
> >> >>>
> >> >>> after this use jps to check if everything is alright or point yo

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

could you post your logs???that would help me in understanding the
problem properly.

Regards,
    Mohammad Tariq


On Thu, Jun 7, 2012 at 1:02 AM, Babak Bastan  wrote:
> Thank you very much mohamad for your attention.I followed the steps but the
> error is the same as the last time.
> and there is my hosts file:
>
> 127.0.0.1       localhost
> #127.0.0.1      ubuntu.ubuntu-domain    ubuntu
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> #::1     ip6-localhost ip6-loopback
> #fe00::0 ip6-localnet
> #ff00::0 ip6-mcastprefix
> #ff02::1 ip6-allnodes
> #ff02::2 ip6-allrouters
>
> but no effect :(
>
> On Wed, Jun 6, 2012 at 8:25 PM, Mohammad Tariq  wrote:
>>
>> also change the permissions of these directories to 777.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Wed, Jun 6, 2012 at 11:54 PM, Mohammad Tariq 
>> wrote:
>> > create a directory "/home/username/hdfs" (or at some place of your
>> > choice)..inside this hdfs directory create three sub directories -
>> > name, data, and temp, then follow these steps :
>> >
>> > add following properties in your core-site.xml -
>> >
>> > 
>> >          fs.default.name
>> >          hdfs://localhost:9000/
>> >        
>> >
>> >        
>> >          hadoop.tmp.dir
>> >          /home/mohammad/hdfs/temp
>> >        
>> >
>> > then add following two properties in your hdfs-site.xml -
>> >
>> > 
>> >                dfs.replication
>> >                1
>> >        
>> >
>> >        
>> >                dfs.name.dir
>> >                /home/mohammad/hdfs/name
>> >        
>> >
>> >        
>> >                dfs.data.dir
>> >                /home/mohammad/hdfs/data
>> >        
>> >
>> > finally add this property in your mapred-site.xml -
>> >
>> >       
>> >          mapred.job.tracker
>> >          hdfs://localhost:9001
>> >        
>> >
>> > NOTE: you can give any name to these directories of your choice, just
>> > keep in mind you have to give same names as values of
>> >           above specified properties in your configuration files.
>> > (give full path of these directories, not just the name of the
>> > directory)
>> >
>> > After this  follow the steps provided in the previous reply.
>> >
>> > Regards,
>> >     Mohammad Tariq
>> >
>> >
>> > On Wed, Jun 6, 2012 at 11:42 PM, Babak Bastan 
>> > wrote:
>> >> thank's Mohammad
>> >>
>> >> with this command:
>> >>
>> >> babak@ubuntu:~/Downloads/hadoop/bin$ hadoop namenode -format
>> >>
>> >> this is my output:
>> >>
>> >> 12/06/06 20:05:20 INFO namenode.NameNode: STARTUP_MSG:
>> >> /
>> >> STARTUP_MSG: Starting NameNode
>> >> STARTUP_MSG:   host = ubuntu/127.0.1.1
>> >> STARTUP_MSG:   args = [-format]
>> >> STARTUP_MSG:   version = 0.20.2
>> >> STARTUP_MSG:   build =
>> >> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
>> >> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
>> >> /
>> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
>> >> fsOwner=babak,babak,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
>> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem: supergroup=supergroup
>> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem: isPermissionEnabled=true
>> >> 12/06/06 20:05:20 INFO common.Storage: Image file of size 95 saved in 0
>> >> seconds.
>> >> 12/06/06 20:05:20 INFO common.Storage: Storage directory
>> >> /tmp/hadoop-babak/dfs/name has been successfully formatted.
>> >> 12/06/06 20:05:20 INFO namenode.NameNode: SHUTDOWN_MSG:
>> >> /
>> >> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
>> >> /
>> >>
>> >> by this command:
>> >>
>> >> babak@ubuntu:~/Downloads/hadoop/bin$ start-dfs.sh
>> >>
>> >> this is the out put
>> >>
>> >> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“
>> >> nicht
>> >> anlegen: Keine Berechtigung
>> >>
>> >> this out put(it's in german and it means no right to make this folder)
>> >>
>> >>
>> >> On Wed, Jun 6, 2012 at 7:59 PM, Mohammad Tariq 
>> >> wrote:
>> >>>
>> >>> once we are done with the configuration, we need to format the file
>> >>> system..use this command to do that-
>> >>> bin/hadoop namenode -format
>> >>>
>> >>> after this, hadoop daemon processes should be started using following
>> >>> commands -
>> >>> bin/start-dfs.sh (it'll start NN & DN)
>> >>> bin/start-mapred.sh (it'll start JT & TT)
>> >>>
>> >>> after this use jps to check if everything is alright or point your
>> >>> browser to localhost:50070..if you further find any problem provide us
>> >>> with the error logs..:)
>> >>>
>> >>> Regards,
>> >>>     Mohammad Tariq
>> >>>
>> >>>
>> >>> On Wed, Jun 6, 2012 at 11:22 PM, Babak Bastan 
>> >>> wrote:
>> >>> > were you able to format hdfs properly???
>> >>> > I did'nt get your question,Do you mean HADOOP_HOME? or where did I
>> >>> > install
>> >>>

Re: Error while Creating Table in Hive

2012-06-06 Thread Babak Bastan

Thank you very much mohamad for your attention.I followed the steps but the
error is the same as the last time.
and there is my hosts file:

127.0.0.1   localhost
#127.0.0.1  ubuntu.ubuntu-domainubuntu


# The following lines are desirable for IPv6 capable hosts

#::1 ip6-localhost ip6-loopback
#fe00::0 ip6-localnet
#ff00::0 ip6-mcastprefix
#ff02::1 ip6-allnodes
#ff02::2 ip6-allrouters

but no effect :(

On Wed, Jun 6, 2012 at 8:25 PM, Mohammad Tariq  wrote:

> also change the permissions of these directories to 777.
>
> Regards,
> Mohammad Tariq
>
>
> On Wed, Jun 6, 2012 at 11:54 PM, Mohammad Tariq 
> wrote:
> > create a directory "/home/username/hdfs" (or at some place of your
> > choice)..inside this hdfs directory create three sub directories -
> > name, data, and temp, then follow these steps :
> >
> > add following properties in your core-site.xml -
> >
> > 
> >  fs.default.name
> >  hdfs://localhost:9000/
> >
> >
> >
> >  hadoop.tmp.dir
> >  /home/mohammad/hdfs/temp
> >
> >
> > then add following two properties in your hdfs-site.xml -
> >
> > 
> >dfs.replication
> >1
> >
> >
> >
> >dfs.name.dir
> >/home/mohammad/hdfs/name
> >
> >
> >
> >dfs.data.dir
> >/home/mohammad/hdfs/data
> >
> >
> > finally add this property in your mapred-site.xml -
> >
> >   
> >  mapred.job.tracker
> >  hdfs://localhost:9001
> >
> >
> > NOTE: you can give any name to these directories of your choice, just
> > keep in mind you have to give same names as values of
> >   above specified properties in your configuration files.
> > (give full path of these directories, not just the name of the
> > directory)
> >
> > After this  follow the steps provided in the previous reply.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Wed, Jun 6, 2012 at 11:42 PM, Babak Bastan 
> wrote:
> >> thank's Mohammad
> >>
> >> with this command:
> >>
> >> babak@ubuntu:~/Downloads/hadoop/bin$ hadoop namenode -format
> >>
> >> this is my output:
> >>
> >> 12/06/06 20:05:20 INFO namenode.NameNode: STARTUP_MSG:
> >> /
> >> STARTUP_MSG: Starting NameNode
> >> STARTUP_MSG:   host = ubuntu/127.0.1.1
> >> STARTUP_MSG:   args = [-format]
> >> STARTUP_MSG:   version = 0.20.2
> >> STARTUP_MSG:   build =
> >> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> >> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> >> /
> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
> >> fsOwner=babak,babak,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem: supergroup=supergroup
> >> 12/06/06 20:05:20 INFO namenode.FSNamesystem: isPermissionEnabled=true
> >> 12/06/06 20:05:20 INFO common.Storage: Image file of size 95 saved in 0
> >> seconds.
> >> 12/06/06 20:05:20 INFO common.Storage: Storage directory
> >> /tmp/hadoop-babak/dfs/name has been successfully formatted.
> >> 12/06/06 20:05:20 INFO namenode.NameNode: SHUTDOWN_MSG:
> >> /
> >> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
> >> /
> >>
> >> by this command:
> >>
> >> babak@ubuntu:~/Downloads/hadoop/bin$ start-dfs.sh
> >>
> >> this is the out put
> >>
> >> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“ nicht
> >> anlegen: Keine Berechtigung
> >>
> >> this out put(it's in german and it means no right to make this folder)
> >>
> >>
> >> On Wed, Jun 6, 2012 at 7:59 PM, Mohammad Tariq 
> wrote:
> >>>
> >>> once we are done with the configuration, we need to format the file
> >>> system..use this command to do that-
> >>> bin/hadoop namenode -format
> >>>
> >>> after this, hadoop daemon processes should be started using following
> >>> commands -
> >>> bin/start-dfs.sh (it'll start NN & DN)
> >>> bin/start-mapred.sh (it'll start JT & TT)
> >>>
> >>> after this use jps to check if everything is alright or point your
> >>> browser to localhost:50070..if you further find any problem provide us
> >>> with the error logs..:)
> >>>
> >>> Regards,
> >>> Mohammad Tariq
> >>>
> >>>
> >>> On Wed, Jun 6, 2012 at 11:22 PM, Babak Bastan 
> wrote:
> >>> > were you able to format hdfs properly???
> >>> > I did'nt get your question,Do you mean HADOOP_HOME? or where did I
> >>> > install
> >>> > Hadoop?
> >>> >
> >>> > On Wed, Jun 6, 2012 at 7:49 PM, Mohammad Tariq 
> >>> > wrote:
> >>> >>
> >>> >> if you are getting only this, it means your hadoop is not
> >>> >> running..were you able to format hdfs properly???
> >>> >>
> >>> >> Regards,
> >>> >> Mohammad Tariq
> >>> >>
> >>> >>
> >>> >> On Wed, Jun 6, 2012 at 11:17 PM, Babak Bas

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

also change the permissions of these directories to 777.

Regards,
    Mohammad Tariq


On Wed, Jun 6, 2012 at 11:54 PM, Mohammad Tariq  wrote:
> create a directory "/home/username/hdfs" (or at some place of your
> choice)..inside this hdfs directory create three sub directories -
> name, data, and temp, then follow these steps :
>
> add following properties in your core-site.xml -
>
> 
>          fs.default.name
>          hdfs://localhost:9000/
>        
>
>        
>          hadoop.tmp.dir
>          /home/mohammad/hdfs/temp
>        
>
> then add following two properties in your hdfs-site.xml -
>
> 
>                dfs.replication
>                1
>        
>
>        
>                dfs.name.dir
>                /home/mohammad/hdfs/name
>        
>
>        
>                dfs.data.dir
>                /home/mohammad/hdfs/data
>        
>
> finally add this property in your mapred-site.xml -
>
>       
>          mapred.job.tracker
>          hdfs://localhost:9001
>        
>
> NOTE: you can give any name to these directories of your choice, just
> keep in mind you have to give same names as values of
>           above specified properties in your configuration files.
> (give full path of these directories, not just the name of the
> directory)
>
> After this  follow the steps provided in the previous reply.
>
> Regards,
>     Mohammad Tariq
>
>
> On Wed, Jun 6, 2012 at 11:42 PM, Babak Bastan  wrote:
>> thank's Mohammad
>>
>> with this command:
>>
>> babak@ubuntu:~/Downloads/hadoop/bin$ hadoop namenode -format
>>
>> this is my output:
>>
>> 12/06/06 20:05:20 INFO namenode.NameNode: STARTUP_MSG:
>> /
>> STARTUP_MSG: Starting NameNode
>> STARTUP_MSG:   host = ubuntu/127.0.1.1
>> STARTUP_MSG:   args = [-format]
>> STARTUP_MSG:   version = 0.20.2
>> STARTUP_MSG:   build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
>> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
>> /
>> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
>> fsOwner=babak,babak,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
>> 12/06/06 20:05:20 INFO namenode.FSNamesystem: supergroup=supergroup
>> 12/06/06 20:05:20 INFO namenode.FSNamesystem: isPermissionEnabled=true
>> 12/06/06 20:05:20 INFO common.Storage: Image file of size 95 saved in 0
>> seconds.
>> 12/06/06 20:05:20 INFO common.Storage: Storage directory
>> /tmp/hadoop-babak/dfs/name has been successfully formatted.
>> 12/06/06 20:05:20 INFO namenode.NameNode: SHUTDOWN_MSG:
>> /
>> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
>> /
>>
>> by this command:
>>
>> babak@ubuntu:~/Downloads/hadoop/bin$ start-dfs.sh
>>
>> this is the out put
>>
>> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“ nicht
>> anlegen: Keine Berechtigung
>>
>> this out put(it's in german and it means no right to make this folder)
>>
>>
>> On Wed, Jun 6, 2012 at 7:59 PM, Mohammad Tariq  wrote:
>>>
>>> once we are done with the configuration, we need to format the file
>>> system..use this command to do that-
>>> bin/hadoop namenode -format
>>>
>>> after this, hadoop daemon processes should be started using following
>>> commands -
>>> bin/start-dfs.sh (it'll start NN & DN)
>>> bin/start-mapred.sh (it'll start JT & TT)
>>>
>>> after this use jps to check if everything is alright or point your
>>> browser to localhost:50070..if you further find any problem provide us
>>> with the error logs..:)
>>>
>>> Regards,
>>>     Mohammad Tariq
>>>
>>>
>>> On Wed, Jun 6, 2012 at 11:22 PM, Babak Bastan  wrote:
>>> > were you able to format hdfs properly???
>>> > I did'nt get your question,Do you mean HADOOP_HOME? or where did I
>>> > install
>>> > Hadoop?
>>> >
>>> > On Wed, Jun 6, 2012 at 7:49 PM, Mohammad Tariq 
>>> > wrote:
>>> >>
>>> >> if you are getting only this, it means your hadoop is not
>>> >> running..were you able to format hdfs properly???
>>> >>
>>> >> Regards,
>>> >>     Mohammad Tariq
>>> >>
>>> >>
>>> >> On Wed, Jun 6, 2012 at 11:17 PM, Babak Bastan 
>>> >> wrote:
>>> >> > Hi MohammadmI irun jps in my shel I can see this result:
>>> >> > 2213 Jps
>>> >> >
>>> >> >
>>> >> > On Wed, Jun 6, 2012 at 7:44 PM, Mohammad Tariq 
>>> >> > wrote:
>>> >> >>
>>> >> >> you can also use "jps" command at your shell to see whether Hadoop
>>> >> >> processes are running or not.
>>> >> >>
>>> >> >> Regards,
>>> >> >>     Mohammad Tariq
>>> >> >>
>>> >> >>
>>> >> >> On Wed, Jun 6, 2012 at 11:12 PM, Mohammad Tariq 
>>> >> >> wrote:
>>> >> >> > Hi Babak,
>>> >> >> >
>>> >> >> >  You have to type it in you web browser..Hadoop provides us a web
>>> >> >> > GUI
>>> >> >> > that not only allows us to browse through the file system, but to
>>> >> >> > download the files as well..Apart from that it also provides a web
>>>

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

create a directory "/home/username/hdfs" (or at some place of your
choice)..inside this hdfs directory create three sub directories -
name, data, and temp, then follow these steps :

add following properties in your core-site.xml -


  fs.default.name
  hdfs://localhost:9000/



  hadoop.tmp.dir
  /home/mohammad/hdfs/temp


then add following two properties in your hdfs-site.xml -


dfs.replication
1



dfs.name.dir
/home/mohammad/hdfs/name



dfs.data.dir
/home/mohammad/hdfs/data


finally add this property in your mapred-site.xml -

   
  mapred.job.tracker
  hdfs://localhost:9001


NOTE: you can give any name to these directories of your choice, just
keep in mind you have to give same names as values of
   above specified properties in your configuration files.
(give full path of these directories, not just the name of the
directory)

After this  follow the steps provided in the previous reply.

Regards,
    Mohammad Tariq


On Wed, Jun 6, 2012 at 11:42 PM, Babak Bastan  wrote:
> thank's Mohammad
>
> with this command:
>
> babak@ubuntu:~/Downloads/hadoop/bin$ hadoop namenode -format
>
> this is my output:
>
> 12/06/06 20:05:20 INFO namenode.NameNode: STARTUP_MSG:
> /
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host = ubuntu/127.0.1.1
> STARTUP_MSG:   args = [-format]
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> /
> 12/06/06 20:05:20 INFO namenode.FSNamesystem:
> fsOwner=babak,babak,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
> 12/06/06 20:05:20 INFO namenode.FSNamesystem: supergroup=supergroup
> 12/06/06 20:05:20 INFO namenode.FSNamesystem: isPermissionEnabled=true
> 12/06/06 20:05:20 INFO common.Storage: Image file of size 95 saved in 0
> seconds.
> 12/06/06 20:05:20 INFO common.Storage: Storage directory
> /tmp/hadoop-babak/dfs/name has been successfully formatted.
> 12/06/06 20:05:20 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
> /
>
> by this command:
>
> babak@ubuntu:~/Downloads/hadoop/bin$ start-dfs.sh
>
> this is the out put
>
> mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“ nicht
> anlegen: Keine Berechtigung
>
> this out put(it's in german and it means no right to make this folder)
>
>
> On Wed, Jun 6, 2012 at 7:59 PM, Mohammad Tariq  wrote:
>>
>> once we are done with the configuration, we need to format the file
>> system..use this command to do that-
>> bin/hadoop namenode -format
>>
>> after this, hadoop daemon processes should be started using following
>> commands -
>> bin/start-dfs.sh (it'll start NN & DN)
>> bin/start-mapred.sh (it'll start JT & TT)
>>
>> after this use jps to check if everything is alright or point your
>> browser to localhost:50070..if you further find any problem provide us
>> with the error logs..:)
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Wed, Jun 6, 2012 at 11:22 PM, Babak Bastan  wrote:
>> > were you able to format hdfs properly???
>> > I did'nt get your question,Do you mean HADOOP_HOME? or where did I
>> > install
>> > Hadoop?
>> >
>> > On Wed, Jun 6, 2012 at 7:49 PM, Mohammad Tariq 
>> > wrote:
>> >>
>> >> if you are getting only this, it means your hadoop is not
>> >> running..were you able to format hdfs properly???
>> >>
>> >> Regards,
>> >>     Mohammad Tariq
>> >>
>> >>
>> >> On Wed, Jun 6, 2012 at 11:17 PM, Babak Bastan 
>> >> wrote:
>> >> > Hi MohammadmI irun jps in my shel I can see this result:
>> >> > 2213 Jps
>> >> >
>> >> >
>> >> > On Wed, Jun 6, 2012 at 7:44 PM, Mohammad Tariq 
>> >> > wrote:
>> >> >>
>> >> >> you can also use "jps" command at your shell to see whether Hadoop
>> >> >> processes are running or not.
>> >> >>
>> >> >> Regards,
>> >> >>     Mohammad Tariq
>> >> >>
>> >> >>
>> >> >> On Wed, Jun 6, 2012 at 11:12 PM, Mohammad Tariq 
>> >> >> wrote:
>> >> >> > Hi Babak,
>> >> >> >
>> >> >> >  You have to type it in you web browser..Hadoop provides us a web
>> >> >> > GUI
>> >> >> > that not only allows us to browse through the file system, but to
>> >> >> > download the files as well..Apart from that it also provides a web
>> >> >> > GUI
>> >> >> > that can be used to see the status of Jobtracker and
>> >> >> > Tasktracker..When
>> >> >> > you run a Hive or Pig job or a Mapreduce job, you can point your
>> >> >> > browser to http://localhost:50030 to see the status and logs of
>> >> >> > your
>> >> >> > job.
>> >> >> >
>> >> >> > Regar

Re: Error while Creating Table in Hive

2012-06-06 Thread Babak Bastan

thank's Mohammad

with this command:

babak@ubuntu:~/Downloads/hadoop/bin$ hadoop namenode -format

this is my output:

12/06/06 20:05:20 INFO namenode.NameNode: STARTUP_MSG:
/
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ubuntu/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
/
12/06/06 20:05:20 INFO namenode.FSNamesystem:
fsOwner=babak,babak,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
12/06/06 20:05:20 INFO namenode.FSNamesystem: supergroup=supergroup
12/06/06 20:05:20 INFO namenode.FSNamesystem: isPermissionEnabled=true
12/06/06 20:05:20 INFO common.Storage: Image file of size 95 saved in 0 seconds.
12/06/06 20:05:20 INFO common.Storage: Storage directory
/tmp/hadoop-babak/dfs/name has been successfully formatted.
12/06/06 20:05:20 INFO namenode.NameNode: SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
/

by this command:

babak@ubuntu:~/Downloads/hadoop/bin$ start-dfs.sh

this is the out put

mkdir: kann Verzeichnis „/home/babak/Downloads/hadoop/bin/../logs“
nicht anlegen: Keine Berechtigung

this out put(it's in german and it means no right to make this folder)

On Wed, Jun 6, 2012 at 7:59 PM, Mohammad Tariq  wrote:

> once we are done with the configuration, we need to format the file
> system..use this command to do that-
> bin/hadoop namenode -format
>
> after this, hadoop daemon processes should be started using following
> commands -
> bin/start-dfs.sh (it'll start NN & DN)
> bin/start-mapred.sh (it'll start JT & TT)
>
> after this use jps to check if everything is alright or point your
> browser to localhost:50070..if you further find any problem provide us
> with the error logs..:)
>
> Regards,
> Mohammad Tariq
>
>
> On Wed, Jun 6, 2012 at 11:22 PM, Babak Bastan  wrote:
> > were you able to format hdfs properly???
> > I did'nt get your question,Do you mean HADOOP_HOME? or where did I
> install
> > Hadoop?
> >
> > On Wed, Jun 6, 2012 at 7:49 PM, Mohammad Tariq 
> wrote:
> >>
> >> if you are getting only this, it means your hadoop is not
> >> running..were you able to format hdfs properly???
> >>
> >> Regards,
> >> Mohammad Tariq
> >>
> >>
> >> On Wed, Jun 6, 2012 at 11:17 PM, Babak Bastan 
> wrote:
> >> > Hi MohammadmI irun jps in my shel I can see this result:
> >> > 2213 Jps
> >> >
> >> >
> >> > On Wed, Jun 6, 2012 at 7:44 PM, Mohammad Tariq 
> >> > wrote:
> >> >>
> >> >> you can also use "jps" command at your shell to see whether Hadoop
> >> >> processes are running or not.
> >> >>
> >> >> Regards,
> >> >> Mohammad Tariq
> >> >>
> >> >>
> >> >> On Wed, Jun 6, 2012 at 11:12 PM, Mohammad Tariq 
> >> >> wrote:
> >> >> > Hi Babak,
> >> >> >
> >> >> >  You have to type it in you web browser..Hadoop provides us a web
> GUI
> >> >> > that not only allows us to browse through the file system, but to
> >> >> > download the files as well..Apart from that it also provides a web
> >> >> > GUI
> >> >> > that can be used to see the status of Jobtracker and
> >> >> > Tasktracker..When
> >> >> > you run a Hive or Pig job or a Mapreduce job, you can point your
> >> >> > browser to http://localhost:50030 to see the status and logs of
> your
> >> >> > job.
> >> >> >
> >> >> > Regards,
> >> >> > Mohammad Tariq
> >> >> >
> >> >> >
> >> >> > On Wed, Jun 6, 2012 at 8:28 PM, Babak Bastan 
> >> >> > wrote:
> >> >> >> Thank you shashwat for the answer,
> >> >> >> where should I type http://localhost:50070?
> >> >> >> I typed here: hive>http://localhost:50070 but nothing as result
> >> >> >>
> >> >> >>
> >> >> >> On Wed, Jun 6, 2012 at 3:32 PM, shashwat shriparv
> >> >> >>  wrote:
> >> >> >>>
> >> >> >>> first type http://localhost:50070 whether this is opening or not
> >> >> >>> and
> >> >> >>> check
> >> >> >>> how many nodes are available, check some of the hadoop shell
> >> >> >>> commands
> >> >> >>> from
> http://hadoop.apache.org/common/docs/r0.18.3/hdfs_shell.html
> >> >> >>> run
> >> >> >>> example mapreduce task on hadoop take example from here
> >> >> >>>
> >> >> >>>
> >> >> >>> :
> http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
> >> >> >>>
> >> >> >>> if all the above you can do sucessfully means hadoop is
> configured
> >> >> >>> correctly
> >> >> >>>
> >> >> >>> Regards
> >> >> >>> Shashwat
> >> >> >>>
> >> >> >>>
> >> >> >>> On Wed, Jun 6, 2012 at 1:30 AM, Babak Bastan  >
> >> >> >>> wrote:
> >> >> 
> >> >>  no I'm not working on CDH.Is there a way to test if my Hadoop
> >> >>  works
> >> >>  fine
> >> >>  or not?
> >> >> 
> >

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Vinod Singh

But it may payoff by saving on network IO while copying the data during
reduce phase. Though it will vary from case to case. We had good results by
using Snappy codec for compressing map output. Snappy provides reasonably
good compression at faster rate.

Thanks,
Vinod

http://blog.vinodsingh.com/

On Wed, Jun 6, 2012 at 4:03 PM, Debarshi Basak wrote:

>  Compression is an overhead when you have a CPU intensive job
>
>
> Debarshi Basak
> Tata Consultancy Services
> Mailto: debarshi.ba...@tcs.com
> Website: http://www.tcs.com
> 
> Experience certainty. IT Services
> Business Solutions
> Outsourcing
> 
>
> -Bejoy Ks ** wrote: -**
>
> To: "user@hive.apache.org" 
> From: Bejoy Ks 
> Date: 06/06/2012 03:37PM
> Subject: Re: Compressed data storage in HDFS - Error
>
>
> Hi Sreenath
>
> Output compression is more useful on storage level, when a larger file is
> compressed it saves on hdfs blocks and there by the cluster become more
> scalable in terms of number of files.
>
> Yes lzo libraries needs to be there in all task tracker nodes as well the
> node that hosts the hive client.
>
> Regards
> Bejoy KS
>
>   --
> *From:* Sreenath Menon 
> *To:* user@hive.apache.org; Bejoy Ks 
> *Sent:* Wednesday, June 6, 2012 3:25 PM
> *Subject:* Re: Compressed data storage in HDFS - Error
>
> Hi Bejoy
> I would like to make this clear.
> There is no gain on processing throughput/time on compressing the data
> stored in HDFS (not talking about intermediate compression)...wright??
> And do I need to add the lzo libraries in Hadoop_Home/lib/native for all
> the nodes (including the slave nodes)??
>
>
>  =-=-=
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

once we are done with the configuration, we need to format the file
system..use this command to do that-
bin/hadoop namenode -format

after this, hadoop daemon processes should be started using following commands -
bin/start-dfs.sh (it'll start NN & DN)
bin/start-mapred.sh (it'll start JT & TT)

after this use jps to check if everything is alright or point your
browser to localhost:50070..if you further find any problem provide us
with the error logs..:)

Regards,
    Mohammad Tariq


On Wed, Jun 6, 2012 at 11:22 PM, Babak Bastan  wrote:
> were you able to format hdfs properly???
> I did'nt get your question,Do you mean HADOOP_HOME? or where did I install
> Hadoop?
>
> On Wed, Jun 6, 2012 at 7:49 PM, Mohammad Tariq  wrote:
>>
>> if you are getting only this, it means your hadoop is not
>> running..were you able to format hdfs properly???
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Wed, Jun 6, 2012 at 11:17 PM, Babak Bastan  wrote:
>> > Hi MohammadmI irun jps in my shel I can see this result:
>> > 2213 Jps
>> >
>> >
>> > On Wed, Jun 6, 2012 at 7:44 PM, Mohammad Tariq 
>> > wrote:
>> >>
>> >> you can also use "jps" command at your shell to see whether Hadoop
>> >> processes are running or not.
>> >>
>> >> Regards,
>> >>     Mohammad Tariq
>> >>
>> >>
>> >> On Wed, Jun 6, 2012 at 11:12 PM, Mohammad Tariq 
>> >> wrote:
>> >> > Hi Babak,
>> >> >
>> >> >  You have to type it in you web browser..Hadoop provides us a web GUI
>> >> > that not only allows us to browse through the file system, but to
>> >> > download the files as well..Apart from that it also provides a web
>> >> > GUI
>> >> > that can be used to see the status of Jobtracker and
>> >> > Tasktracker..When
>> >> > you run a Hive or Pig job or a Mapreduce job, you can point your
>> >> > browser to http://localhost:50030 to see the status and logs of your
>> >> > job.
>> >> >
>> >> > Regards,
>> >> >     Mohammad Tariq
>> >> >
>> >> >
>> >> > On Wed, Jun 6, 2012 at 8:28 PM, Babak Bastan 
>> >> > wrote:
>> >> >> Thank you shashwat for the answer,
>> >> >> where should I type http://localhost:50070?
>> >> >> I typed here: hive>http://localhost:50070 but nothing as result
>> >> >>
>> >> >>
>> >> >> On Wed, Jun 6, 2012 at 3:32 PM, shashwat shriparv
>> >> >>  wrote:
>> >> >>>
>> >> >>> first type http://localhost:50070 whether this is opening or not
>> >> >>> and
>> >> >>> check
>> >> >>> how many nodes are available, check some of the hadoop shell
>> >> >>> commands
>> >> >>> from http://hadoop.apache.org/common/docs/r0.18.3/hdfs_shell.html
>> >> >>> run
>> >> >>> example mapreduce task on hadoop take example from here
>> >> >>>
>> >> >>>
>> >> >>> : http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
>> >> >>>
>> >> >>> if all the above you can do sucessfully means hadoop is configured
>> >> >>> correctly
>> >> >>>
>> >> >>> Regards
>> >> >>> Shashwat
>> >> >>>
>> >> >>>
>> >> >>> On Wed, Jun 6, 2012 at 1:30 AM, Babak Bastan 
>> >> >>> wrote:
>> >> 
>> >>  no I'm not working on CDH.Is there a way to test if my Hadoop
>> >>  works
>> >>  fine
>> >>  or not?
>> >> 
>> >> 
>> >>  On Tue, Jun 5, 2012 at 9:55 PM, Bejoy KS 
>> >>  wrote:
>> >> >
>> >> > Hi Babak
>> >> >
>> >> > You gotta follow those instructions in the apace site to set up
>> >> > hadoop
>> >> > from scratch and ensure that hdfs is working first. You should be
>> >> > able to
>> >> > read and write files to hdfs before you do your next steps.
>> >> >
>> >> > Are you on CDH or apache distribution of hadoop? If it is CDH
>> >> > there
>> >> > are
>> >> > detailed instructions on Cloudera web site.
>> >> >
>> >> > Regards
>> >> > Bejoy KS
>> >> >
>> >> > Sent from handheld, please excuse typos.
>> >> > 
>> >> > From: Babak Bastan 
>> >> > Date: Tue, 5 Jun 2012 21:30:22 +0200
>> >> > To: 
>> >> > ReplyTo: user@hive.apache.org
>> >> > Subject: Re: Error while Creating Table in Hive
>> >> >
>> >> > @Bejoy: I set the fs.default.name in the core-site.xml and I did
>> >> > all
>> >> > of
>> >> > thing that was mentioned in the reference but no effect
>> >> >
>> >> > On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan 
>> >> > wrote:
>> >> >>
>> >> >> Ok sorry but that was my Mistake .I thought it works but no.
>> >> >> I wrote the command without ; and then I think It works but with
>> >> >> ;
>> >> >> at
>> >> >> the end of command
>> >> >>
>> >> >> CREATE TABLE pokes (foo INT, bar STRING);
>> >> >>
>> >> >> does'nt work
>> >> >>
>> >> >>
>> >> >> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv
>> >> >>  wrote:
>> >> >>>
>> >> >>> inside configuration. all properties will be inside the
>> >> >>> configuration
>> >> >>> tags
>> >> >>>
>> >> >>>
>> >> >

Re: Error while Creating Table in Hive

2012-06-06 Thread Babak Bastan

*were you able to format hdfs properly???*
I did'nt get your question,Do you mean HADOOP_HOME? or where did I install
Hadoop?

On Wed, Jun 6, 2012 at 7:49 PM, Mohammad Tariq  wrote:

> if you are getting only this, it means your hadoop is not
> running..were you able to format hdfs properly???
>
> Regards,
> Mohammad Tariq
>
>
> On Wed, Jun 6, 2012 at 11:17 PM, Babak Bastan  wrote:
> > Hi MohammadmI irun jps in my shel I can see this result:
> > 2213 Jps
> >
> >
> > On Wed, Jun 6, 2012 at 7:44 PM, Mohammad Tariq 
> wrote:
> >>
> >> you can also use "jps" command at your shell to see whether Hadoop
> >> processes are running or not.
> >>
> >> Regards,
> >> Mohammad Tariq
> >>
> >>
> >> On Wed, Jun 6, 2012 at 11:12 PM, Mohammad Tariq 
> >> wrote:
> >> > Hi Babak,
> >> >
> >> >  You have to type it in you web browser..Hadoop provides us a web GUI
> >> > that not only allows us to browse through the file system, but to
> >> > download the files as well..Apart from that it also provides a web GUI
> >> > that can be used to see the status of Jobtracker and Tasktracker..When
> >> > you run a Hive or Pig job or a Mapreduce job, you can point your
> >> > browser to http://localhost:50030 to see the status and logs of your
> >> > job.
> >> >
> >> > Regards,
> >> > Mohammad Tariq
> >> >
> >> >
> >> > On Wed, Jun 6, 2012 at 8:28 PM, Babak Bastan 
> wrote:
> >> >> Thank you shashwat for the answer,
> >> >> where should I type http://localhost:50070?
> >> >> I typed here: hive>http://localhost:50070 but nothing as result
> >> >>
> >> >>
> >> >> On Wed, Jun 6, 2012 at 3:32 PM, shashwat shriparv
> >> >>  wrote:
> >> >>>
> >> >>> first type http://localhost:50070 whether this is opening or not
> and
> >> >>> check
> >> >>> how many nodes are available, check some of the hadoop shell
> commands
> >> >>> from http://hadoop.apache.org/common/docs/r0.18.3/hdfs_shell.html
> run
> >> >>> example mapreduce task on hadoop take example from here
> >> >>>
> >> >>> :
> http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
> >> >>>
> >> >>> if all the above you can do sucessfully means hadoop is configured
> >> >>> correctly
> >> >>>
> >> >>> Regards
> >> >>> Shashwat
> >> >>>
> >> >>>
> >> >>> On Wed, Jun 6, 2012 at 1:30 AM, Babak Bastan 
> >> >>> wrote:
> >> 
> >>  no I'm not working on CDH.Is there a way to test if my Hadoop works
> >>  fine
> >>  or not?
> >> 
> >> 
> >>  On Tue, Jun 5, 2012 at 9:55 PM, Bejoy KS 
> wrote:
> >> >
> >> > Hi Babak
> >> >
> >> > You gotta follow those instructions in the apace site to set up
> >> > hadoop
> >> > from scratch and ensure that hdfs is working first. You should be
> >> > able to
> >> > read and write files to hdfs before you do your next steps.
> >> >
> >> > Are you on CDH or apache distribution of hadoop? If it is CDH
> there
> >> > are
> >> > detailed instructions on Cloudera web site.
> >> >
> >> > Regards
> >> > Bejoy KS
> >> >
> >> > Sent from handheld, please excuse typos.
> >> > 
> >> > From: Babak Bastan 
> >> > Date: Tue, 5 Jun 2012 21:30:22 +0200
> >> > To: 
> >> > ReplyTo: user@hive.apache.org
> >> > Subject: Re: Error while Creating Table in Hive
> >> >
> >> > @Bejoy: I set the fs.default.name in the core-site.xml and I did
> all
> >> > of
> >> > thing that was mentioned in the reference but no effect
> >> >
> >> > On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan 
> >> > wrote:
> >> >>
> >> >> Ok sorry but that was my Mistake .I thought it works but no.
> >> >> I wrote the command without ; and then I think It works but with
> ;
> >> >> at
> >> >> the end of command
> >> >>
> >> >> CREATE TABLE pokes (foo INT, bar STRING);
> >> >>
> >> >> does'nt work
> >> >>
> >> >>
> >> >> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv
> >> >>  wrote:
> >> >>>
> >> >>> inside configuration. all properties will be inside the
> >> >>> configuration
> >> >>> tags
> >> >>>
> >> >>>
> >> >>> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan <
> babak...@gmail.com>
> >> >>> wrote:
> >> 
> >>  Thank you so much my friend your idee works fine(no error) you
> >>  are
> >>  the best :)
> >> 
> >> 
> >>  On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan <
> babak...@gmail.com>
> >>  wrote:
> >> >
> >> > It must be inside the  or
> outside
> >> > this?
> >> >
> >> >
> >> > On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv
> >> >  wrote:
> >> >>
> >> >> It will be inside hive/conf
> >> >>
> >> >>
> >> >> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan
> >> >> 
> >> >> wrote:
> >> >>>
>

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

if you are getting only this, it means your hadoop is not
running..were you able to format hdfs properly???

Regards,
    Mohammad Tariq


On Wed, Jun 6, 2012 at 11:17 PM, Babak Bastan  wrote:
> Hi MohammadmI irun jps in my shel I can see this result:
> 2213 Jps
>
>
> On Wed, Jun 6, 2012 at 7:44 PM, Mohammad Tariq  wrote:
>>
>> you can also use "jps" command at your shell to see whether Hadoop
>> processes are running or not.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>> On Wed, Jun 6, 2012 at 11:12 PM, Mohammad Tariq 
>> wrote:
>> > Hi Babak,
>> >
>> >  You have to type it in you web browser..Hadoop provides us a web GUI
>> > that not only allows us to browse through the file system, but to
>> > download the files as well..Apart from that it also provides a web GUI
>> > that can be used to see the status of Jobtracker and Tasktracker..When
>> > you run a Hive or Pig job or a Mapreduce job, you can point your
>> > browser to http://localhost:50030 to see the status and logs of your
>> > job.
>> >
>> > Regards,
>> >     Mohammad Tariq
>> >
>> >
>> > On Wed, Jun 6, 2012 at 8:28 PM, Babak Bastan  wrote:
>> >> Thank you shashwat for the answer,
>> >> where should I type http://localhost:50070?
>> >> I typed here: hive>http://localhost:50070 but nothing as result
>> >>
>> >>
>> >> On Wed, Jun 6, 2012 at 3:32 PM, shashwat shriparv
>> >>  wrote:
>> >>>
>> >>> first type http://localhost:50070 whether this is opening or not and
>> >>> check
>> >>> how many nodes are available, check some of the hadoop shell commands
>> >>> from http://hadoop.apache.org/common/docs/r0.18.3/hdfs_shell.html  run
>> >>> example mapreduce task on hadoop take example from here
>> >>>
>> >>> : http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
>> >>>
>> >>> if all the above you can do sucessfully means hadoop is configured
>> >>> correctly
>> >>>
>> >>> Regards
>> >>> Shashwat
>> >>>
>> >>>
>> >>> On Wed, Jun 6, 2012 at 1:30 AM, Babak Bastan 
>> >>> wrote:
>> 
>>  no I'm not working on CDH.Is there a way to test if my Hadoop works
>>  fine
>>  or not?
>> 
>> 
>>  On Tue, Jun 5, 2012 at 9:55 PM, Bejoy KS  wrote:
>> >
>> > Hi Babak
>> >
>> > You gotta follow those instructions in the apace site to set up
>> > hadoop
>> > from scratch and ensure that hdfs is working first. You should be
>> > able to
>> > read and write files to hdfs before you do your next steps.
>> >
>> > Are you on CDH or apache distribution of hadoop? If it is CDH there
>> > are
>> > detailed instructions on Cloudera web site.
>> >
>> > Regards
>> > Bejoy KS
>> >
>> > Sent from handheld, please excuse typos.
>> > 
>> > From: Babak Bastan 
>> > Date: Tue, 5 Jun 2012 21:30:22 +0200
>> > To: 
>> > ReplyTo: user@hive.apache.org
>> > Subject: Re: Error while Creating Table in Hive
>> >
>> > @Bejoy: I set the fs.default.name in the core-site.xml and I did all
>> > of
>> > thing that was mentioned in the reference but no effect
>> >
>> > On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan 
>> > wrote:
>> >>
>> >> Ok sorry but that was my Mistake .I thought it works but no.
>> >> I wrote the command without ; and then I think It works but with ;
>> >> at
>> >> the end of command
>> >>
>> >> CREATE TABLE pokes (foo INT, bar STRING);
>> >>
>> >> does'nt work
>> >>
>> >>
>> >> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv
>> >>  wrote:
>> >>>
>> >>> inside configuration. all properties will be inside the
>> >>> configuration
>> >>> tags
>> >>>
>> >>>
>> >>> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan 
>> >>> wrote:
>> 
>>  Thank you so much my friend your idee works fine(no error) you
>>  are
>>  the best :)
>> 
>> 
>>  On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan 
>>  wrote:
>> >
>> > It must be inside the  or outside
>> > this?
>> >
>> >
>> > On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv
>> >  wrote:
>> >>
>> >> It will be inside hive/conf
>> >>
>> >>
>> >> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan
>> >> 
>> >> wrote:
>> >>>
>> >>> Thanks sShashwat, and where is this hive-site.xml
>> >>>
>> >>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv
>> >>>  wrote:
>> 
>>  set
>> 
>>  hive.metastore.warehouse.dir in hive-site.xml
>> 
>>  
>>    hive.metastore.local
>>    true
>>  
>> 
>>  hive.metastore.warehouse.dir
>>                 /home/>  username>/hivefolder
>> >

Re: Error while Creating Table in Hive

2012-06-06 Thread Babak Bastan

Hi MohammadmI irun jps in my shel I can see this result:
2213 Jps


On Wed, Jun 6, 2012 at 7:44 PM, Mohammad Tariq  wrote:

> you can also use "jps" command at your shell to see whether Hadoop
> processes are running or not.
>
> Regards,
> Mohammad Tariq
>
>
> On Wed, Jun 6, 2012 at 11:12 PM, Mohammad Tariq 
> wrote:
> > Hi Babak,
> >
> >  You have to type it in you web browser..Hadoop provides us a web GUI
> > that not only allows us to browse through the file system, but to
> > download the files as well..Apart from that it also provides a web GUI
> > that can be used to see the status of Jobtracker and Tasktracker..When
> > you run a Hive or Pig job or a Mapreduce job, you can point your
> > browser to http://localhost:50030 to see the status and logs of your
> > job.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Wed, Jun 6, 2012 at 8:28 PM, Babak Bastan  wrote:
> >> Thank you shashwat for the answer,
> >> where should I type http://localhost:50070?
> >> I typed here: hive>http://localhost:50070 but nothing as result
> >>
> >>
> >> On Wed, Jun 6, 2012 at 3:32 PM, shashwat shriparv
> >>  wrote:
> >>>
> >>> first type http://localhost:50070 whether this is opening or not and
> check
> >>> how many nodes are available, check some of the hadoop shell commands
> >>> from http://hadoop.apache.org/common/docs/r0.18.3/hdfs_shell.html  run
> >>> example mapreduce task on hadoop take example from here
> >>> :
> http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
> >>>
> >>> if all the above you can do sucessfully means hadoop is configured
> >>> correctly
> >>>
> >>> Regards
> >>> Shashwat
> >>>
> >>>
> >>> On Wed, Jun 6, 2012 at 1:30 AM, Babak Bastan 
> wrote:
> 
>  no I'm not working on CDH.Is there a way to test if my Hadoop works
> fine
>  or not?
> 
> 
>  On Tue, Jun 5, 2012 at 9:55 PM, Bejoy KS  wrote:
> >
> > Hi Babak
> >
> > You gotta follow those instructions in the apace site to set up
> hadoop
> > from scratch and ensure that hdfs is working first. You should be
> able to
> > read and write files to hdfs before you do your next steps.
> >
> > Are you on CDH or apache distribution of hadoop? If it is CDH there
> are
> > detailed instructions on Cloudera web site.
> >
> > Regards
> > Bejoy KS
> >
> > Sent from handheld, please excuse typos.
> > 
> > From: Babak Bastan 
> > Date: Tue, 5 Jun 2012 21:30:22 +0200
> > To: 
> > ReplyTo: user@hive.apache.org
> > Subject: Re: Error while Creating Table in Hive
> >
> > @Bejoy: I set the fs.default.name in the core-site.xml and I did
> all of
> > thing that was mentioned in the reference but no effect
> >
> > On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan 
> wrote:
> >>
> >> Ok sorry but that was my Mistake .I thought it works but no.
> >> I wrote the command without ; and then I think It works but with ;
> at
> >> the end of command
> >>
> >> CREATE TABLE pokes (foo INT, bar STRING);
> >>
> >> does'nt work
> >>
> >>
> >> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv
> >>  wrote:
> >>>
> >>> inside configuration. all properties will be inside the
> configuration
> >>> tags
> >>>
> >>>
> >>> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan 
> >>> wrote:
> 
>  Thank you so much my friend your idee works fine(no error) you are
>  the best :)
> 
> 
>  On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan 
>  wrote:
> >
> > It must be inside the  or outside
> > this?
> >
> >
> > On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv
> >  wrote:
> >>
> >> It will be inside hive/conf
> >>
> >>
> >> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan <
> babak...@gmail.com>
> >> wrote:
> >>>
> >>> Thanks sShashwat, and where is this hive-site.xml
> >>>
> >>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv
> >>>  wrote:
> 
>  set
> 
>  hive.metastore.warehouse.dir in hive-site.xml
> 
>  
>    hive.metastore.local
>    true
>  
> 
>  hive.metastore.warehouse.dir
> /home//hivefolder
> location of default database for
> the
>  warehouse
> 
> 
> 
>  On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan
>   wrote:
> >
> > Hello Experts ,
> >
> > I'm new in Hive .When try to create a test Table in Hive I
> get
> > an error.I want to run this command:
> >

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

you can also use "jps" command at your shell to see whether Hadoop
processes are running or not.

Regards,
    Mohammad Tariq


On Wed, Jun 6, 2012 at 11:12 PM, Mohammad Tariq  wrote:
> Hi Babak,
>
>  You have to type it in you web browser..Hadoop provides us a web GUI
> that not only allows us to browse through the file system, but to
> download the files as well..Apart from that it also provides a web GUI
> that can be used to see the status of Jobtracker and Tasktracker..When
> you run a Hive or Pig job or a Mapreduce job, you can point your
> browser to http://localhost:50030 to see the status and logs of your
> job.
>
> Regards,
>     Mohammad Tariq
>
>
> On Wed, Jun 6, 2012 at 8:28 PM, Babak Bastan  wrote:
>> Thank you shashwat for the answer,
>> where should I type http://localhost:50070?
>> I typed here: hive>http://localhost:50070 but nothing as result
>>
>>
>> On Wed, Jun 6, 2012 at 3:32 PM, shashwat shriparv
>>  wrote:
>>>
>>> first type http://localhost:50070 whether this is opening or not and check
>>> how many nodes are available, check some of the hadoop shell commands
>>> from http://hadoop.apache.org/common/docs/r0.18.3/hdfs_shell.html  run
>>> example mapreduce task on hadoop take example from here
>>> : http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
>>>
>>> if all the above you can do sucessfully means hadoop is configured
>>> correctly
>>>
>>> Regards
>>> Shashwat
>>>
>>>
>>> On Wed, Jun 6, 2012 at 1:30 AM, Babak Bastan  wrote:

 no I'm not working on CDH.Is there a way to test if my Hadoop works fine
 or not?


 On Tue, Jun 5, 2012 at 9:55 PM, Bejoy KS  wrote:
>
> Hi Babak
>
> You gotta follow those instructions in the apace site to set up hadoop
> from scratch and ensure that hdfs is working first. You should be able to
> read and write files to hdfs before you do your next steps.
>
> Are you on CDH or apache distribution of hadoop? If it is CDH there are
> detailed instructions on Cloudera web site.
>
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
> 
> From: Babak Bastan 
> Date: Tue, 5 Jun 2012 21:30:22 +0200
> To: 
> ReplyTo: user@hive.apache.org
> Subject: Re: Error while Creating Table in Hive
>
> @Bejoy: I set the fs.default.name in the core-site.xml and I did all of
> thing that was mentioned in the reference but no effect
>
> On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan  wrote:
>>
>> Ok sorry but that was my Mistake .I thought it works but no.
>> I wrote the command without ; and then I think It works but with ; at
>> the end of command
>>
>> CREATE TABLE pokes (foo INT, bar STRING);
>>
>> does'nt work
>>
>>
>> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv
>>  wrote:
>>>
>>> inside configuration. all properties will be inside the configuration
>>> tags
>>>
>>>
>>> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan 
>>> wrote:

 Thank you so much my friend your idee works fine(no error) you are
 the best :)


 On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan 
 wrote:
>
> It must be inside the  or outside
> this?
>
>
> On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv
>  wrote:
>>
>> It will be inside hive/conf
>>
>>
>> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan 
>> wrote:
>>>
>>> Thanks sShashwat, and where is this hive-site.xml
>>>
>>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv
>>>  wrote:

 set

 hive.metastore.warehouse.dir in hive-site.xml

 
   hive.metastore.local
   true
 

 hive.metastore.warehouse.dir
                /home//hivefolder
                location of default database for the
 warehouse
        


 On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan
  wrote:
>
> Hello Experts ,
>
> I'm new in Hive .When try to create a test Table in Hive I get
> an error.I want to run this command:
> CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);
> but this error occured:
> FAILED: Error in metadata: MetaException(message:Got exception:
> java.io.FileNotFoundException File file:/user/hive/warehouse/test 
> does not
> exist.)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> How can I solve this Problem?
>>

Re: Error while Creating Table in Hive

2012-06-06 Thread Mohammad Tariq

Hi Babak,

  You have to type it in you web browser..Hadoop provides us a web GUI
that not only allows us to browse through the file system, but to
download the files as well..Apart from that it also provides a web GUI
that can be used to see the status of Jobtracker and Tasktracker..When
you run a Hive or Pig job or a Mapreduce job, you can point your
browser to http://localhost:50030 to see the status and logs of your
job.

Regards,
    Mohammad Tariq


On Wed, Jun 6, 2012 at 8:28 PM, Babak Bastan  wrote:
> Thank you shashwat for the answer,
> where should I type http://localhost:50070?
> I typed here: hive>http://localhost:50070 but nothing as result
>
>
> On Wed, Jun 6, 2012 at 3:32 PM, shashwat shriparv
>  wrote:
>>
>> first type http://localhost:50070 whether this is opening or not and check
>> how many nodes are available, check some of the hadoop shell commands
>> from http://hadoop.apache.org/common/docs/r0.18.3/hdfs_shell.html  run
>> example mapreduce task on hadoop take example from here
>> : http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
>>
>> if all the above you can do sucessfully means hadoop is configured
>> correctly
>>
>> Regards
>> Shashwat
>>
>>
>> On Wed, Jun 6, 2012 at 1:30 AM, Babak Bastan  wrote:
>>>
>>> no I'm not working on CDH.Is there a way to test if my Hadoop works fine
>>> or not?
>>>
>>>
>>> On Tue, Jun 5, 2012 at 9:55 PM, Bejoy KS  wrote:

 Hi Babak

 You gotta follow those instructions in the apace site to set up hadoop
 from scratch and ensure that hdfs is working first. You should be able to
 read and write files to hdfs before you do your next steps.

 Are you on CDH or apache distribution of hadoop? If it is CDH there are
 detailed instructions on Cloudera web site.

 Regards
 Bejoy KS

 Sent from handheld, please excuse typos.
 
 From: Babak Bastan 
 Date: Tue, 5 Jun 2012 21:30:22 +0200
 To: 
 ReplyTo: user@hive.apache.org
 Subject: Re: Error while Creating Table in Hive

 @Bejoy: I set the fs.default.name in the core-site.xml and I did all of
 thing that was mentioned in the reference but no effect

 On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan  wrote:
>
> Ok sorry but that was my Mistake .I thought it works but no.
> I wrote the command without ; and then I think It works but with ; at
> the end of command
>
> CREATE TABLE pokes (foo INT, bar STRING);
>
> does'nt work
>
>
> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv
>  wrote:
>>
>> inside configuration. all properties will be inside the configuration
>> tags
>>
>>
>> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan 
>> wrote:
>>>
>>> Thank you so much my friend your idee works fine(no error) you are
>>> the best :)
>>>
>>>
>>> On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan 
>>> wrote:

 It must be inside the  or outside
 this?


 On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv
  wrote:
>
> It will be inside hive/conf
>
>
> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan 
> wrote:
>>
>> Thanks sShashwat, and where is this hive-site.xml
>>
>> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv
>>  wrote:
>>>
>>> set
>>>
>>> hive.metastore.warehouse.dir in hive-site.xml
>>>
>>> 
>>>   hive.metastore.local
>>>   true
>>> 
>>>
>>> hive.metastore.warehouse.dir
>>>/home//hivefolder
>>>location of default database for the
>>> warehouse
>>>
>>>
>>>
>>> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan
>>>  wrote:

 Hello Experts ,

 I'm new in Hive .When try to create a test Table in Hive I get
 an error.I want to run this command:
 CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);
 but this error occured:
 FAILED: Error in metadata: MetaException(message:Got exception:
 java.io.FileNotFoundException File file:/user/hive/warehouse/test 
 does not
 exist.)
 FAILED: Execution Error, return code 1 from
 org.apache.hadoop.hive.ql.exec.DDLTask
 How can I solve this Problem?
 Thank you so much
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> ∞
>>>
>>> Shashwat Shriparv
>>>
>>>
>>
>
>
>
> --
>
>
> ∞
>
> Shashwat Shriparv
>
>

Re: Error while Creating Table in Hive

2012-06-06 Thread Babak Bastan

Thank you shashwat for the answer,
where should I type http://localhost:50070?
I typed here: hive>http://localhost:50070 but nothing as result

On Wed, Jun 6, 2012 at 3:32 PM, shashwat shriparv  wrote:

> first type http://localhost:50070 whether this is opening or not and
> check how many nodes are available, check some of the hadoop shell commands
> from http://hadoop.apache.org/common/docs/r0.18.3/hdfs_shell.html  run
> example mapreduce task on hadoop take example from here :
> http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
>
> if all the above you can do sucessfully means hadoop is configured
> correctly
>
> Regards
> Shashwat
>
>
> On Wed, Jun 6, 2012 at 1:30 AM, Babak Bastan  wrote:
>
>> no I'm not working on CDH.Is there a way to test if my Hadoop works fine
>> or not?
>>
>>
>> On Tue, Jun 5, 2012 at 9:55 PM, Bejoy KS  wrote:
>>
>>> **
>>> Hi Babak
>>>
>>> You gotta follow those instructions in the apace site to set up hadoop
>>> from scratch and ensure that hdfs is working first. You should be able to
>>> read and write files to hdfs before you do your next steps.
>>>
>>> Are you on CDH or apache distribution of hadoop? If it is CDH there are
>>> detailed instructions on Cloudera web site.
>>>
>>> Regards
>>> Bejoy KS
>>>
>>> Sent from handheld, please excuse typos.
>>> --
>>> *From: * Babak Bastan 
>>> *Date: *Tue, 5 Jun 2012 21:30:22 +0200
>>> *To: *
>>> *ReplyTo: * user@hive.apache.org
>>> *Subject: *Re: Error while Creating Table in Hive
>>>
>>> @Bejoy: I set the fs.default.name in the core-site.xml and I did all of
>>> thing that was mentioned in the reference but no effect
>>>
>>> On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan  wrote:
>>>
 Ok sorry but that was my Mistake .I thought it works but no.
 I wrote the command without ; and then I think It works but with ; at
 the end of command

 CREATE TABLE pokes (foo INT, bar STRING);

 does'nt work


 On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv <
 dwivedishash...@gmail.com> wrote:

> inside configuration. all properties will be inside the configuration
> tags
>
>
> On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan wrote:
>
>> Thank you so much my friend your idee works fine(no error) you are
>> the best :)
>>
>>
>> On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan wrote:
>>
>>> It must be inside the  or outside
>>> this?
>>>
>>>
>>> On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv <
>>> dwivedishash...@gmail.com> wrote:
>>>
 It will be inside hive/conf


 On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan 
 wrote:

> Thanks sShashwat, and where is this hive-site.xml
>
> On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
> dwivedishash...@gmail.com> wrote:
>
>> set
>>
>> hive.metastore.warehouse.dir in hive-site.xml
>>
>> 
>>   hive.metastore.local
>>   true
>> 
>>
>> hive.metastore.warehouse.dir
>>/home//hivefolder
>>location of default database for the
>> warehouse
>>
>>
>>
>> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan > > wrote:
>>
>>> Hello Experts ,
>>>
>>> I'm new in Hive .When try to create a test Table in Hive I get
>>> an error.I want to run this command:
>>> *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
>>> but this error occured:
>>> FAILED: Error in metadata: MetaException(message:Got exception:
>>> java.io.FileNotFoundException File file:/user/hive/warehouse/test 
>>> does not
>>> exist.)
>>> FAILED: Execution Error, return code 1 from
>>> org.apache.hadoop.hive.ql.exec.DDLTask
>>> How can I solve this Problem?
>>> Thank you so much
>>>
>>
>>
>>
>> --
>>
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>>
>


 --


 ∞
 Shashwat Shriparv



>>>
>>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>

>>>
>>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>

Re: Error while Creating Table in Hive

2012-06-06 Thread shashwat shriparv

first type http://localhost:50070 whether this is opening or not and check
how many nodes are available, check some of the hadoop shell commands from
http://hadoop.apache.org/common/docs/r0.18.3/hdfs_shell.html  run example
mapreduce task on hadoop take example from here :
http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/

if all the above you can do sucessfully means hadoop is configured correctly

Regards
Shashwat

On Wed, Jun 6, 2012 at 1:30 AM, Babak Bastan  wrote:

> no I'm not working on CDH.Is there a way to test if my Hadoop works fine
> or not?
>
>
> On Tue, Jun 5, 2012 at 9:55 PM, Bejoy KS  wrote:
>
>> **
>> Hi Babak
>>
>> You gotta follow those instructions in the apace site to set up hadoop
>> from scratch and ensure that hdfs is working first. You should be able to
>> read and write files to hdfs before you do your next steps.
>>
>> Are you on CDH or apache distribution of hadoop? If it is CDH there are
>> detailed instructions on Cloudera web site.
>>
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>> --
>> *From: * Babak Bastan 
>> *Date: *Tue, 5 Jun 2012 21:30:22 +0200
>> *To: *
>> *ReplyTo: * user@hive.apache.org
>> *Subject: *Re: Error while Creating Table in Hive
>>
>> @Bejoy: I set the fs.default.name in the core-site.xml and I did all of
>> thing that was mentioned in the reference but no effect
>>
>> On Tue, Jun 5, 2012 at 8:43 PM, Babak Bastan  wrote:
>>
>>> Ok sorry but that was my Mistake .I thought it works but no.
>>> I wrote the command without ; and then I think It works but with ; at
>>> the end of command
>>>
>>> CREATE TABLE pokes (foo INT, bar STRING);
>>>
>>> does'nt work
>>>
>>>
>>> On Tue, Jun 5, 2012 at 8:34 PM, shashwat shriparv <
>>> dwivedishash...@gmail.com> wrote:
>>>
 inside configuration. all properties will be inside the configuration
 tags


 On Tue, Jun 5, 2012 at 11:53 PM, Babak Bastan wrote:

> Thank you so much my friend your idee works fine(no error) you are the
> best :)
>
>
> On Tue, Jun 5, 2012 at 8:20 PM, Babak Bastan wrote:
>
>> It must be inside the  or outside this?
>>
>>
>> On Tue, Jun 5, 2012 at 8:15 PM, shashwat shriparv <
>> dwivedishash...@gmail.com> wrote:
>>
>>> It will be inside hive/conf
>>>
>>>
>>> On Tue, Jun 5, 2012 at 11:43 PM, Babak Bastan wrote:
>>>
 Thanks sShashwat, and where is this hive-site.xml

 On Tue, Jun 5, 2012 at 8:02 PM, shashwat shriparv <
 dwivedishash...@gmail.com> wrote:

> set
>
> hive.metastore.warehouse.dir in hive-site.xml
>
> 
>   hive.metastore.local
>   true
> 
>
> hive.metastore.warehouse.dir
>/home//hivefolder
>location of default database for the
> warehouse
>
>
>
> On Tue, Jun 5, 2012 at 10:43 PM, Babak Bastan 
> wrote:
>
>> Hello Experts ,
>>
>> I'm new in Hive .When try to create a test Table in Hive I get an
>> error.I want to run this command:
>> *CREATE TABLE Test (DateT STRING, Url STRING, Content STRING);*
>> but this error occured:
>> FAILED: Error in metadata: MetaException(message:Got exception:
>> java.io.FileNotFoundException File file:/user/hive/warehouse/test 
>> does not
>> exist.)
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.DDLTask
>> How can I solve this Problem?
>> Thank you so much
>>
>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>

>>>
>>>
>>> --
>>>
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>>
>>
>


 --


 ∞
 Shashwat Shriparv



>>>
>>
>


-- 


∞
Shashwat Shriparv

Re: Importing data into Hive

2012-06-06 Thread Jagat Singh

One of similar use case which I worked in , the record timestamp is not
guaranteed to arrive in some order. So we used Pig to do some processing
similar to what your custom code is doing and after the records are in
required order of timestamp we push them to hive.

---
Sent from Mobile , short and crisp.
On 06-Jun-2012 4:50 PM, "Philip Tromans"  wrote:

> Hi all,
>
> I'm interested in knowing how everyone is importing their data into
> their production Hive clusters.
>
> Let me explain a little more. At the moment, I have log files (which
> are divided into 5 minute chunks, per event type (of which there are
> around 10), per server (a few 10s) arriving on one of my Hadoop nodes.
> These log files then get glued together by some custom code, into
> entity+hour buckets. The code does a few non-utterly trivial things:
> * It supports an (almost) atomic file append.
> * It parses the timestamp out of each line in the log file to ensure
> that it ends up in the correct hour bucket (because some log file
> rotation ends up with some events from x:04:59.9 being written in the
> wrong file).
> * Once an entity+hour bucket hasn't changed for a while, it gets
> pushed into Hive.
>
> There's a bug in the code which is proving hard to track down amongst
> our high volume logs (hundreds of millions to billions of events per
> day), and we're going to shortly replace this architecture anyway with
> something based around Kafka/Flume/Storm, but I need an interim
> solution to the log aggregation problem.
>
> I can't just load the raw log files into Hive because it will most
> probably make the metastore grind to a halt (we easily processes tens
> of thousands of log files per day), and having one map task per file
> (I think) means the overheads of processing all of these files will be
> non-trivial. Is something like Flume the right way to go?
>
> If people are happy to share their data import strategies then that'd be
> great.
>
> Cheers,
>
> Phil.
>

Importing data into Hive

2012-06-06 Thread Philip Tromans

Hi all,

I'm interested in knowing how everyone is importing their data into
their production Hive clusters.

Let me explain a little more. At the moment, I have log files (which
are divided into 5 minute chunks, per event type (of which there are
around 10), per server (a few 10s) arriving on one of my Hadoop nodes.
These log files then get glued together by some custom code, into
entity+hour buckets. The code does a few non-utterly trivial things:
* It supports an (almost) atomic file append.
* It parses the timestamp out of each line in the log file to ensure
that it ends up in the correct hour bucket (because some log file
rotation ends up with some events from x:04:59.9 being written in the
wrong file).
* Once an entity+hour bucket hasn't changed for a while, it gets
pushed into Hive.

There's a bug in the code which is proving hard to track down amongst
our high volume logs (hundreds of millions to billions of events per
day), and we're going to shortly replace this architecture anyway with
something based around Kafka/Flume/Storm, but I need an interim
solution to the log aggregation problem.

I can't just load the raw log files into Hive because it will most
probably make the metastore grind to a halt (we easily processes tens
of thousands of log files per day), and having one map task per file
(I think) means the overheads of processing all of these files will be
non-trivial. Is something like Flume the right way to go?

If people are happy to share their data import strategies then that'd be great.

Cheers,

Phil.

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Debarshi Basak

 Compression is an overhead when you have a CPU intensive jobDebarshi BasakTata Consultancy ServicesMailto: debarshi.ba...@tcs.comWebsite: http://www.tcs.comExperience certainty.	IT Services			Business Solutions			Outsourcing-Bejoy Ks  wrote: -To: "user@hive.apache.org" From: Bejoy Ks Date: 06/06/2012 03:37PMSubject: Re: Compressed data storage in HDFS - ErrorHi SreenathOutput compression is more useful on storage level, when a larger file is compressed it saves on hdfs blocks and there by the cluster become more scalable in terms of number of files. Yes lzo libraries needs to be there in all task tracker nodes as well the node that hosts the hive client.RegardsBejoy KS  From: Sreenath Menon  To: user@hive.apache.org; Bejoy Ks   Sent: Wednesday, June 6, 2012 3:25 PM Subject: Re: Compressed data storage in HDFS - Error   
Hi BejoyI would like to make this clear.There is no gain on processing throughput/time on compressing the data stored in HDFS (not talking about intermediate compression)...wright??And do I need to add the lzo libraries in Hadoop_Home/lib/native for all the nodes (including the slave nodes)??

=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Bejoy Ks



Hi Sreenath

Output compression is more useful on storage level, when a larger file is 
compressed it saves on hdfs blocks and there by the cluster become more 
scalable in terms of number of files. 

Yes lzo libraries needs to be there in all task tracker nodes as well the node 
that hosts the hive client.

Regards
Bejoy KS




 From: Sreenath Menon 
To: user@hive.apache.org; Bejoy Ks  
Sent: Wednesday, June 6, 2012 3:25 PM
Subject: Re: Compressed data storage in HDFS - Error
 

Hi Bejoy
I would like to make this clear.
There is no gain on processing throughput/time on compressing the data stored 
in HDFS (not talking about intermediate compression)...wright??
And do I need to add the lzo libraries in Hadoop_Home/lib/native for all the 
nodes (including the slave nodes)??

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Sreenath Menon

Hi Bejoy
I would like to make this clear.
There is no gain on processing throughput/time on compressing the data
stored in HDFS (not talking about intermediate compression)...wright??
And do I need to add the lzo libraries in Hadoop_Home/lib/native for all
the nodes (including the slave nodes)??

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Bejoy Ks

Hi Sreenath


The default compression codec used in hadoop is
org.apache.hadoop.io.compress.DefaultCodec

To use gzip as compression
mapred.output.compress=truemapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec


Regards
Bejoy KS





 From: Sreenath Menon 
To: user@hive.apache.org 
Sent: Wednesday, June 6, 2012 3:08 PM
Subject: Re: Compressed data storage in HDFS - Error
 

Thanks for the response.
1)How do I use the Gz compression and does it come with Hadoop. Or else how do 
I build a compression method for using in Hive. I would like to run evaluation 
across compression methods.
What is the default compression used in Hadoop.


2)Kindly bear with me if this question is stupid. I am not talking about
 compression within intermediate steps. Storing the raw data in 
compressed format, how can this be useful since data needs to be decompressed 
for executing a job...wright?.

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Bejoy Ks

Hi Sreenath

The lzo error is because you don't have the lzo libraries in 
Hadoop_Home/lib/native folder. You need to pack/build lzo for the OS you are 
using.

In compression as you mentioned there is an overhead in decompressing while 
processing the records. HDFS is used to store large amount of data so 
compression saves much on storage space (consider replication as well). Now it 
is not final output compression that speeds up map reduce jobs but it the 
intermediate compression that has this advantage. Intermediate compression 
means compression of map output. In a map reduce job there is much of copy and 
shuffle happening between the map and reduce phases, when this intermediate 
data is compressed this operation is faster as it consumes much lesser IO. 


The following properties enables intermediate compression
mapred.compress.map.output=true
mapred.map.output.compression.codec= hadoop.compression.lzo.LzoCodec


Regards
Bejoy KS




 From: Siddharth Tiwari 
To: "user@hive.apache.org "  
Sent: Wednesday, June 6, 2012 2:58 PM
Subject: RE: Compressed data storage in HDFS - Error
 

There is something you gain and something you loose.
Compression would reduce IO through increased cpu work . Also you would receive 
different experience for different tasks ie HDFS read , HDFS write , shuffle 
and sort . So to go for compression or not depends on your usages .
Sent from my N8




-Original Message- 
From: Sreenath Menon 
Sent: 6/6/2012 8:50:23 AM 
To: user@hive.apache.org 
Subject: Compressed data storage in HDFS - Error 
I would like to compress my data in the HDFS using some Hive commands.
Step followed: (data already residing in table sample)

create table rc_lzo like sample;
SET hive.exec.compress.output=true;
SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;
insert overwrite table rc_lzo select * from sample;

Error:
Compression codec com\.hadoop\.compression\.lzo\.LzoCodec was not found

1)What do I need to do to use Lzo as well as other compression methods?

2)Heard somewhere that :Using compressed data will produce better results than 
uncompressed data in some cases. How can this be, as there is always a 
compression and decompression time allotted with compression methods. Any truth 
in this, if so how ? Can understand
 how there are better results when using compression between 
mappers-to-reducers and in between map-reduce jobs.

Thanks and Regards
Sreenath Mullassery

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Sreenath Menon

k...understood...so you load the compressed data into memory (thereby
decreasing the size of file needed to be loaded) and then apply
decompression algorithm to get the uncompressed data. is this what happens?

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Sreenath Menon

Thanks for the response.
1)How do I use the Gz compression and does it come with Hadoop. Or else how
do I build a compression method for using in Hive. I would like to run
evaluation across compression methods.
What is the default compression used in Hadoop.


2)Kindly bear with me if this question is stupid. I am not talking about
compression within intermediate steps. Storing the raw data in compressed
format, how can this be useful since data needs to be decompressed for
executing a job...wright?.

RE: Compressed data storage in HDFS - Error

2012-06-06 Thread Siddharth Tiwari

There is something you gain and something you loose.
Compression would reduce IO through increased cpu work . Also you would receive 
different experience for different tasks ie HDFS read , HDFS write , shuffle 
and sort . So to go for compression or not depends on your usages .
Sent from my N8



-Original Message-
From: Sreenath Menon
Sent: 6/6/2012 8:50:23 AM
To: user@hive.apache.org
Subject: Compressed data storage in HDFS - Error
I would like to compress my data in the HDFS using some Hive commands.
Step followed: (data already residing in table sample)

create table rc_lzo like sample;
SET hive.exec.compress.output=true;
SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;
insert overwrite table rc_lzo select * from sample;

Error:
Compression codec com\.hadoop\.compression\.lzo\.LzoCodec was not found

1)What do I need to do to use Lzo as well as other compression methods?

2)Heard somewhere that :Using compressed data will produce better results than 
uncompressed data in some cases. How can this be, as there is always a 
compression and decompression time allotted with compression methods. Any truth 
in this, if so how ? Can understand how there are better results when using 
compression between mappers-to-reducers and in between map-reduce jobs.

Thanks and Regards
Sreenath Mullassery

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Debarshi Basak

Basically, when your data is compressed you have lesser IO than your uncompressd data. During job execution is doesn't decompress. It would be a relevant question in Hadoop's mailing list than hive.Debarshi BasakTata Consultancy ServicesMailto: debarshi.ba...@tcs.comWebsite: http://www.tcs.comExperience certainty. IT ServicesBusiness SolutionsOutsourcing-Sreenath Menon wrote: - 

To: user@hive.apache.orgFrom: Sreenath Menon Date: 06/06/2012 02:20PMSubject: Compressed data storage in HDFS - ErrorI would like to compress my data in the HDFS using some Hive commands.Step followed: (data already residing in table sample)create table rc_lzo like sample;SET hive.exec.compress.output=true;SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;insert overwrite table rc_lzo select * from sample;Error:Compression codec com\.hadoop\.compression\.lzo\.LzoCodec was not found1)What do I need to do to use Lzo as well as other compression methods?2)Heard somewhere that :Using compressed data will produce better results than uncompressed data in some cases. How can this be, as there is always a compression and decompression time allotted with compression methods. Any truth in this, if so how ? Can understand how there are better results when using compression between mappers-to-reducers and in between map-reduce jobs.Thanks and RegardsSreenath Mullassery
=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Debarshi Basak

Yes performance is better because your IO is less when your data is lessDebarshi BasakTata Consultancy ServicesMailto: debarshi.ba...@tcs.comWebsite: http://www.tcs.comExperience certainty. IT ServicesBusiness SolutionsOutsourcing-Sreenath Menon wrote: - 

To: user@hive.apache.orgFrom: Sreenath Menon Date: 06/06/2012 02:20PMSubject: Compressed data storage in HDFS - ErrorI would like to compress my data in the HDFS using some Hive commands.Step followed: (data already residing in table sample)create table rc_lzo like sample;SET hive.exec.compress.output=true;SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;insert overwrite table rc_lzo select * from sample;Error:Compression codec com\.hadoop\.compression\.lzo\.LzoCodec was not found1)What do I need to do to use Lzo as well as other compression methods?2)Heard somewhere that :Using compressed data will produce better results than uncompressed data in some cases. How can this be, as there is always a compression and decompression time allotted with compression methods. Any truth in this, if so how ? Can understand how there are better results when using compression between mappers-to-reducers and in between map-reduce jobs.Thanks and RegardsSreenath Mullassery
=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

Re: Compressed data storage in HDFS - Error

2012-06-06 Thread Debarshi Basak

LZO doesn't ship with apache hadoop you need to build it..try GZDebarshi BasakTata Consultancy ServicesMailto: debarshi.ba...@tcs.comWebsite: http://www.tcs.comExperience certainty. IT ServicesBusiness SolutionsOutsourcing-Sreenath Menon wrote: - 

To: user@hive.apache.orgFrom: Sreenath Menon Date: 06/06/2012 02:20PMSubject: Compressed data storage in HDFS - ErrorI would like to compress my data in the HDFS using some Hive commands.Step followed: (data already residing in table sample)create table rc_lzo like sample;SET hive.exec.compress.output=true;SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;insert overwrite table rc_lzo select * from sample;Error:Compression codec com\.hadoop\.compression\.lzo\.LzoCodec was not found1)What do I need to do to use Lzo as well as other compression methods?2)Heard somewhere that :Using compressed data will produce better results than uncompressed data in some cases. How can this be, as there is always a compression and decompression time allotted with compression methods. Any truth in this, if so how ? Can understand how there are better results when using compression between mappers-to-reducers and in between map-reduce jobs.Thanks and RegardsSreenath Mullassery
=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

Re: Front end visualization tool with Hive (when using as a warehouse)

2012-06-06 Thread Sreenath Menon

Thanks all
All help is greatly appreciated. Pl. feel free to post whatever comes to
your mind.
Learned a lot from this conversation.
Pl. post any findings on this topic : Hive as a warehouse - limitations

Thanks

Re: How to install hive?

2012-06-06 Thread Jagat

Hello Rafel,

I assume that you have donwloaded and configured Hadoop successfully. If no
, then please tell i can give steps for that also.

For Hive , the easiest way for me is to download the Hive Tar from Apache
website
Extract the Hive tar into some location
Then i would set environment variables e.g

e.g export HIVE_HOME="/home/hadoop/software/hive-0.9.0"
export PATH=$PATH:$HIVE_HOME/bin

Assuming HADOOP_HOME is already set in your computer , if not set it.

Also If i am going to do concurrent sessions in HIVE i would also configure
hive mysql 
databasefor
storing hive metadata

That's it

Please let me know if you need any detailed help.

Thanks,

Jagat Singh

On Wed, Jun 6, 2012 at 3:42 AM, Rafael Maffud Carlini
wrote:

> Hello everyone, I develop a scientific research for my college, where
> I conduct experiments involving hive and I wonder what is the easiest
> way to install the hive.
>
> I've tried installing using claudera, but I had trouble picking up the
> mirrors.
>
>  I'd rather install it using ubuntu or other debian distribution. In
> my case, I need to work on Hadoop sigle-node cluster.
>
> I appreciate everyone's attention.
>

Re: Multi-group-by select always scans entire table

2012-06-06 Thread Jan Dolinár

Hi Mark,

Thanks for all your help. I tried to run a series of test with various
settings of hive.optimize.ppd and various queries ( see it here
http://pastebin.com/E89p9Ubx ) and now I'm even more confused than
before. In all cases, regardless if the WHERE clause asks about
partitioned or regular column, the result with ppd=true and ppd=false
differ only in file paths, but the structure is the same. Even if I
run the query without the LATERAL VIEW...

Either there is something terribly wrong with hive and/or my setup
and/or I'm completely dumb. Do I understand it right that ppd should
push common criteria from where clauses into earlier stage, so that
there is less data and processing in the following stages? I'm quite
convinced it doesn't really happen here...

Jan

On 6/6/12, Mark Grover  wrote:
> Hi Jan,
> The quick answer is I don't know but may be someone else on the mailing
> list does:-)
>
> Looking at the wiki page for Lateral view(
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView),
> there was a problem related to predicate pushdown on UDTF's (
> https://issues.apache.org/jira/browse/HIVE-1056). However, that seemed to
> have been fixed in Hive 0.6.0 so it shouldn't have any impact on you.
>
> The fix for above ticket introduced a unit test (at
> ql/src/test/results/clientpositive/lateral_view_ppd.q) that tests predicate
> pushdown on UDTF's. Now, all the subsequent releases should have had that
> test pass (otherwise they wouldn't have been released, I hope). The test
> checks for a non-partition column for predicate pushdown. I wonder if it
> makes a difference with a partition column being used.
>
> Can you verify if your query with predicate pushdown enabled works as
> expected with a non-partition column in the where clause? In that case, the
> explain/explain extended output should be different from when predicate
> pushdown is disabled. If predicate pushdown works for non-partition columns
> but not for partition columns, please create a JIRA stating that predicate
> pushdown on UDTF's doesn't work with partition columns.
>
> If it doesn't work for both partition and non-partition columns, then
> obviously Hive-1056 is not working for you. We can take it up on the
> mailing list from there.
>
> Thanks for your input, Jan.
>
> Mark
>
> On Tue, Jun 5, 2012 at 1:19 AM, Jan Dolinár  wrote:
>
>>
>>
>> On Mon, Jun 4, 2012 at 7:20 PM, Mark Grover  wrote:
>>
>>> Hi Jan,
>>> Glad you found something workable.
>>>
>>> What version of Hive are you using? Could you also please check what the
>>> value of the property hive.optimize.ppd is for you?
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>> Hi Mark,
>>
>> Thanks for reply. I'm using hive 0.7.1 distributed from Cloudera as
>> cdh3u4. The property hive.optimize.ppd is set true, but I have tried to
>> turn it off and it doesn't effect the behavior of the problematic query
>> at
>> all. Any other ideas? :-)
>>
>> Also could some of you good guys try to check this on hadoop 0.8 or
>> newer?
>> It would be nice to know if it is worth to go through all the hassle of
>> upgrading or if it won't help. Also, if it is not fixed already, it might
>> be good idea to report it as a bug.
>>
>> Jan
>>
>

37 matches

Mail list logo