Recovering the namenode from failure

I'm on CDH4, and trying to recover both the namenode and cloudera manager VMs from HDFS after losing the namenode. All of our backup VMs are on HDFS, so for the moment I just want to hack something together, copy the backup VMs off HDFS and get on with properly reconfiguring via CDH Manger.

Re: Flume port issue

Just a friendly follow up to see if anyone has any suggestions for the issue with port given below. Any help is appreciated. Thanks Sai On May 20, 2013 5:40 PM, "Sai Sai" wrote: Not sure if this is the right group to ask questions about flume: > > >I am getting an exception about unable to open

Re: Unable to stop Thrift Server

Raj Which version r u using ? I think from 0.9+ onwards its best to use service to stop and start and NOT hive sudo service hive-metastore stop sudo service hive-server stop sudo service hive-metastore start sudo service hive-server start Couple of general things that might help 1. Use linux s

Keep Kerberos credentials valid after logging out

Hi all, I would like to run my hadoop job in a bash file for several times, e.g. #!/usr/bin/env bash for i in {1..10} do my-hadoop-job done Since I don't want to keep my laptop on for hours, I run this bash script on a server via a SSH session. However, the bash script always ter

Re: Keep Kerberos credentials valid after logging out

nohup ./your_bash 1>temp.log 2>&1 & --Send from my Sony mobile. On May 21, 2013 6:32 PM, "zheyi rong" wrote: > Hi all, > > I would like to run my hadoop job in a bash file for several times, e.g. > #!/usr/bin/env bash > for i in {1..10} > do > my-hadoop-job > done > > Since I d

Re: Keep Kerberos credentials valid after logging out

I think you can have a keytab file for the user and use that for authentication. It would renew the credentials when it expires. On Tue, May 21, 2013 at 4:01 PM, zheyi rong wrote: > Hi all, > > I would like to run my hadoop job in a bash file for several times, e.g. > #!/usr/bin/env bash > for

Project ideas

Hello fellow users, We are a group of students studying in National University of Singapore. As part of our course curriculum we need to develop an application using Hadoop and map-reduce. Can you please suggest some innovative ideas for our project? Thanks in advance. Anshuman

Re: Project ideas

Hi Anshuman, Since MR is like: split the input, map it to different node, run it in parallel, and combine the result. I would suggest you look into the application of the Divide-and-Conquer algorithms, and port it, or rewrite it in Hadoop MapReduce. yours, Ling Kun On Tue, May 21, 2013 at

Re: Project ideas

Drink heavily? Sorry. Let me rephrase. Part of the exercise is for you, the student to come up with the idea. Not solicit someone else for a suggestion. This is how you learn. The exercise is to get you to think about the following: 1) What is Hadoop 2) How does it work 3) Why would you wa

ETL Tools

Hello users, I am interested in hearing about what sort of ETL tools are you using with your cloud based apps. Ideally, I am looking ETL(s) with the following feature: -free (yup) -open-source/community support -handles different types of sources or atleast has plugins may be (email, rss, filesys

Viewing snappy compressed files

Hey, there. My Google skills have failed me, and I hope someone here can point me in the right direction. We're storing data on our Hadoop cluster in Snappy compressed format. When we pull a raw file down and try to read it, however, the Snappy libraries don't know how to read the files. They

Re: Flume port issue

This could be a myriad of things, but some points to note: 1. Flume has its own mailing list - I've moved your post there instead. 2. Your command line suggests connecting to localhost:4000, but the log shows ubuntu:41414 instead (different attempts maybe?) 3. The correct type for file roll sink

Re: Flume port issue

Reattempting to post this to the flume ML after subscribing... On 21 May 2013 16:29, Paul Wilkinson wrote: > This could be a myriad of things, but some points to note: > > 1. Flume has its own mailing list - I've moved your post there instead. > 2. Your command line suggests connecting to local

Re: ETL Tools

Apache Flume is one option. On Tue, May 21, 2013 at 7:32 AM, Aji Janis wrote: > Hello users, > > I am interested in hearing about what sort of ETL tools are you using with > your cloud based apps. Ideally, I am looking ETL(s) with the following > feature: > > -free (yup) > -open-source/communit

Re: ETL Tools

For batch imports, I would also suggest Sqoop. Very easy to use, specially if you have mySql into the play. I have not used Sqoop 2 but that is suppose to add enterprise level robustness and admin support as well. -Shahab On Tue, May 21, 2013 at 12:17 PM, Peyman Mohajerian wrote: > Apache Flume

Re: Recovering the namenode from failure

Hi David, You shouldn't need to do anything to get your DNs to report in -- as best they can tell, it's the same NN. Do you see any error messages in the DN logs? -Todd On Tue, May 21, 2013 at 12:30 AM, David Parks wrote: > I’m on CDH4, and trying to recover both the namenode and cloudera manag

Re: ETL Tools

Hello, Flume is better,more sophisticated one is Pentaho, it open source but you have to pay for support. On Tue, May 21, 2013 at 9:52 PM, Shahab Yunus wrote: > For batch imports, I would also suggest Sqoop. Very easy to use, specially > if you have mySql into the play. I have not used Sqoop 2

Re: Recovering the namenode from failure

I think what he's missing is to change the configurations to point to the new name node. It sounds like the new NN has a different IP address from the old NN so the DNs don't know who to report to... On May 21, 2013, at 11:23 AM, Todd Lipcon wrote: > Hi David, > > You shouldn't need to do

Re: Viewing snappy compressed files

Hi Robert, What command are you using to extract your data from hadoop? JM Hey, there. My Google skills have failed me, and I hope someone here can point me in the right direction. ** ** We’re storing data on our Hadoop cluster in Snappy compressed format. When we pull a raw file down and

Re: Project ideas

You want to add any new simple feature to Hadoop or develop an application using hadoop. Sometime back another university student wanted to add encryption to HDFS.Its just a pointer. Just a problem which might interest your university. Talk to the IT dept of NSU and collect as much as server log

Re: Viewing snappy compressed files

If you have snappy codec in io.compression.codecs then you can easily decompress the data out of hdfs directly with a simple command. hadoop fs -text Regards Bejoy KS Sent from remote device, Please excuse typos -Original Message- From: Jean-Marc Spaggiari Date: Tue, 21 May 2013 12:

Re: Viewing snappy compressed files

I haven't tried this with snappy , but you can try using hadoop fs -text On Tue, May 21, 2013 at 8:28 PM, Robert Rapplean < robert.rappl...@trueffect.com> wrote: > Hey, there. My Google skills have failed me, and I hope someone here can > point me in the right direction. > > ** ** > > We’r

MapReduce shuffle algorithm

I am very interested in a deep understanding of the MapReduce "Shuffle" phase algorithm and implementation. Are there whitepapers I could read for an explanation? Or another mailing list for this question? Obviously there is the code ;-) john

hive.metastore.warehouse.dir - Should it point to a physical directory

Hi,   I am configurinig Hive. I ahve a question on the property hive.metastore.warehouse.dir.   Should this point to a physical directory. I am guessing it is a logical directory under Hadoop fs.default.name. Please advise whether I need to create any directory for the variable hive.metastore.wa

Re: Keep Kerberos credentials valid after logging out

Thanks for answering. but the same exceptions appeared: 13/05/21 19:07:01 ERROR security.UserGroupInformation: PriviledgedActionException as:s117449 (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level

Re: Keep Kerberos credentials valid after logging out

could you please give me a bit more details? I am really not good at kerberoes. Thanks. Regards, Zheyi Rong On Tue, May 21, 2013 at 3:34 PM, Rahul Bhattacharjee < rahul.rec@gmail.com> wrote: > I think you can have a keytab file for the user and use that for > authentication. It would renew

Re: Project ideas

Here's a nice link to get started http://hadoopblog.blogspot.in/2010/11/hadoop-research-topics.html Regards Niranjan Singh On Tue, May 21, 2013 at 10:20 PM, Rahul Bhattacharjee < rahul.rec@gmail.com> wrote: > You want to add any new simple feature to Hadoop or develop an application > usin

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

Can some one help me on this ? I am stuck installing and configuring Hive with Oracle. Your timely help is really aprreciated. From: Raj Hadoop To: Hive ; User Sent: Tuesday, May 21, 2013 1:08 PM Subject: hive.metastore.warehouse.dir - Should it point to a ph

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

Ok.I got it. My questions -   1) Should a local physical directory be created before using this property? 2) Should a HDFS file directory be created from Hadoop before using this property?     From: Dean Wampler To: u...@hive.apache.org; Raj Hadoop Cc: User

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

Notes below From: Raj Hadoop mailto:hadoop...@yahoo.com>> Reply-To: "u...@hive.apache.org" mailto:u...@hive.apache.org>>, Raj Hadoop mailto:hadoop...@yahoo.com>> Date: Tuesday, May 21, 2013 10:49 AM To: Dean Wampler mailto:deanwamp...@gmail.com>>, "u...@hive.apache.

Re: Viewing snappy compressed files

+1 Thanks Rahul-da Or u can use hdfs dfs -text /path/to/dir/on/hdfs/part-r-0.snappy | less From: Rahul Bhattacharjee mailto:rahul.rec@gmail.com>> Reply-To: "user@hadoop.apache.org" mailto:user@hadoop.apache.org>> Date: Tuesday, May 21, 2013 9:52 AM To: "u

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

Hi, If by "local physical directory" you mean a directory in the underlying OS file system, then No. You just need to create a directory in HDFS and ad it to that xml config file. Thanks, On Tue, May 21, 2013 at 11:19 PM, Raj Hadoop wrote: > Ok.I got it. My questions - > > 1) Should a local p

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

Thanks Sanjay.   My environment is  like this.   $ echo $HADOOP_HOME /software/home/hadoop/hadoop/hadoop-1.1.2   $ echo $HIVE_HOME /software/home/hadoop/hive/hive-0.9.0 $ id uid=50052(hadoop) gid=600(apps) groups=600(apps)   So can i do like this:   $pwd /software/home/hadoop/hive/hive-0.9.0   $m

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

yes thats what i meant. local physical directory. thanks. From: bharath vissapragada To: u...@hive.apache.org; Raj Hadoop Cc: User Sent: Tuesday, May 21, 2013 1:59 PM Subject: Re: hive.metastore.warehouse.dir - Should it point to a physical directory Hi,

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

So that means I need to create a HDFS ( Not an OS physical directory ) directory under Hadoop that need to be used in the Hive config file for this property. Right? From: Dean Wampler To: Raj Hadoop Cc: Sanjay Subramanian ; "u...@hive.apache.org" ; User S

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

Yes ! On Tue, May 21, 2013 at 11:41 PM, Raj Hadoop wrote: > So that means I need to create a HDFS ( Not an OS physical directory ) > directory under Hadoop that need to be used in the Hive config file for > this property. Right? > > *From:* Dean Wampler > *To:* Raj Hadoop > *Cc:* Sanjay Subr

Re: Project ideas

+1 My $0.02 is look look around and see problems u can solve…Its better to get a list of problems and see if u can model a solution using map-reduce framework An example is as follows PROBLEM Build a Cars Pricing Model based on advertisements on Craigs list OBJECTIVE Recommend a price to the C

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

Thanks Sanjay From: Sanjay Subramanian To: bharath vissapragada ; "u...@hive.apache.org" ; Raj Hadoop Cc: User Sent: Tuesday, May 21, 2013 2:27 PM Subject: Re: hive.metastore.warehouse.dir - Should it point to a physical directory Hi Raj http://www.clo

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

Hi Raj http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Quick-Start/cdh4qs_topic_3.html Installing CDH4 on a Single Linux Node in Pseudo-distributed Mode On the left panel of the page u will find info on Hive installation etc. I suggest CHD4 distribution only beca

Re: Low latency data access Vs High throughput of data

Hi Chris, hi Raj, in relational databases there are different targets for the optimizer: Return the first record as fast as possible, even if reading through the whole dataset takes longer (low latency) Return all rows as fast as possible, even if reading the first record may take a longer time (h

HDFS append overhead

I am trying to determine if it is feasible for multiple nodes to alternate appends to a shared file in HDFS. Can someone tell me, what is the overhead of an open/append/close cycle? If multiple nodes attempt open-for-append at once, the losers queue nicely waiting for the winner to close? --john

Shuffle phase replication factor

When MapReduce enters "shuffle" to partition the tuples, I am assuming that it writes intermediate data to HDFS. What replication factor is used for those temporary files? john

Re: Shuffle phase replication factor

The map output doesn't get written to HDFS. The map task writes its output to its local disk, the reduce tasks will pull the data through HTTP for further processing. Am 21.05.2013 um 19:57 schrieb John Lilley : > When MapReduce enters “shuffle” to partition the tuples, I am assuming that > it

Re: Shuffle phase replication factor

Intermediate data is written to local disk, not to HDFS. Ian. On May 21, 2013, at 1:57 PM, John Lilley wrote: > When MapReduce enters “shuffle” to partition the tuples, I am assuming that > it writes intermediate data to HDFS. What replication factor is used for > those temporary files? > jo

Where to get Oracle scripts for Hive Metastore

I am trying to get Oracle scripts for Hive Metastore.   http://mail-archives.apache.org/mod_mbox/hive-commits/201204.mbox/%3c20120423201303.9742b2388...@eris.apache.org%3E   The scripts in the above link has a  + at the begining of each line. How should I supposed to execute scripts like this thro

Re: Where to get Oracle scripts for Hive Metastore

I got it. This is the link.   http://svn.apache.org/viewvc/hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.9.0.oracle.sql?revision=1329416&view=co&pathrev=1329416 From: Raj Hadoop To: Hive ; User Sent: Tuesday, May 21, 2013 3:08 PM Subject: Where to g

Re: Where to get Oracle scripts for Hive Metastore

Raj The correct location of the script is where u deflated the hive tar For example /usr/lib/hive/scripts/metastore/upgrade/oracle You will find a file in this directory called hive-schema-0.9.0.oracle.sql Use this sanjay From: Raj Hadoop mailto:hadoop...@yahoo.com>> Reply-To: "user@hadoop.ap

Re: Where to get Oracle scripts for Hive Metastore

Sanjay -   This is the first location I tried. But Apache Hive 0.9.0 doesnt have an oracle folder. It only had mysql and derby.   Thanks, Raj From: Sanjay Subramanian To: "user@hadoop.apache.org" ; Raj Hadoop ; Hive Sent: Tuesday, May 21, 2013 3:12 PM Subject

Re: hive.metastore.warehouse.dir - Should it point to a physical directory

No, you only need a directory in HDFS, which will be "virtually located" somewhere in your cluster automatically by HDFS. Also there's a typo in your hive.xml: Should be /correct/path/in/hdfs/to/your/warehouse/directory On Tue, May 21, 2013 at 1:04 PM, Raj Hadoop wrote: > Thanks Sanjay.

LZO compression implementation in Hive

Hi Programming Hive Book authors Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as o

Re: Unable to stop Thrift Server

Not that I know of…..sorry sanjay From: Raj Hadoop mailto:hadoop...@yahoo.com>> Reply-To: Raj Hadoop mailto:hadoop...@yahoo.com>> Date: Monday, May 20, 2013 2:17 PM To: Sanjay Subramanian mailto:sanjay.subraman...@wizecommerce.com>>, "u...@hive.apache.org" mailto:u.

Re: MapReduce shuffle algorithm

An introduction to the subject can be found in the best known reference : Hadoop: The Definitive Guide, 3rd Edition Storage and Analysis at Internet Scale By Tom White Publisher: O'Reilly Media / Yahoo Press Released: May 2012 Chapter 6

Re: Hive on Oracle

Hi The procedure is same as setting up mysql metastore. You need to use the jdbc driver/jar corresponding to the oracle version/release you are intending to use. Regards Bejoy KS Sent from remote device, Please excuse typos -Original Message- From: Raj Hadoop Date: Fri, 17 May 2013 1

Re: Hive on Oracle

Hi Raj Which jar depends on what version of oracle you are using? The jar version corresponding to each oracle release would be there in oracle documentations online. JDBC Jars should be available from the oracle website for free download. Regards Bejoy KS Sent from remote device, Please ex

Re: Where to get Oracle scripts for Hive Metastore

I think it should be this link because this refers to the /branches/branch-0.9 http://svn.apache.org/viewvc/hive/branches/branch-0.9/metastore/scripts/upgrade/oracle/ Can one of the Hive committers please verify…thanks sanjay From: Raj Hadoop mailto:hadoop...@yahoo.com>> Reply-To: "u...@hive

ORA-01950: no privileges on tablespace

  I am setting up a metastore on Oracle for Hive. I executed the script hive-schema-0.9.0-sql file too succesfully.   When i ran this hive > show tables;   I am getting the following error.   ORA-01950: no privileges on tablespace   What kind of Oracle privileges are required (Quota wise for Hive)

Re: ORA-01950: no privileges on tablespace

See the CDH notes here…scroll down to where the Oracle section is http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_18_4.html From: Raj Hadoop mailto:hadoop...@yahoo.com>> Reply-To: "u...@hive.apache.org"

Re: ETL Tools

Thanks for the suggestion. What about Clover or Talend? Have any of you tried it before - interested in knowing how it compares against Pentaho On Tue, May 21, 2013 at 12:26 PM, sudhakara st wrote: > Hello, > > Flume is better,more sophisticated one is Pentaho, it open source but you > have to p

Is there a way to limit # of hadoop tasks per user at runtime?

Hi Everyone, I was wondering if there is a way for limiting the number of tasks (map+reduce) *per user* at runtime? Using an environment variable perhaps? I am asking this from a resource provisioning perspective. I am trying to come up with a N-token licensing system for multiple users to use

Re: Is there a way to limit # of hadoop tasks per user at runtime?

Hi Mehmet, Are you using MR1 or MR2? The fair scheduler, present in both versions, but configured slightly differently, allows you to limit the number of map and reduce tasks in a queue. The configuration can be updated at runtime by modifying the scheduler's allocations file. It also has a fea

upgrade from primary + secondary NN to active + standby NN

Hi guys, I am planning to upgrade to HA architecture and I am not sure if metadata is backward capable. Are there any problems that anybody is aware of? Thanks in advance. Serge

heartbeat and timeout question

Hello Hadoopers, I am going to migrate production racks of datanodes/tasktrackers into new core switches. Rack awareness is in place for long time. I am looking for the way to mitigate recopying blocks of datanodes in the rack that is being move (when it become dead nodes), and shifting of running

Re: Project ideas

Excellent Sanjay, really excellent input. Many Thanks for this input. I have been always thinking about some ideas but never knowing what to proceed with. Thanks again. Sai From: Sanjay Subramanian To: "user@hadoop.apache.org" Sent: Tuesday, 21 May 2013 11:51

Re: Project ideas

im a newbie but maybe this will also add some value... it is my understanding that mapreduce is like a distributed "group by" statement when you run a statement like this against your petabyes of dataset it can take a long time.. first and foremost because the first thing you have to do before you

Re: Recovering the namenode from failure

I think he's mentioned the new NN is the same IP and Hostname as the old one, and uses an actual checkpoint. All he has to do is start the DNs back up again and they should report in fine. On Tue, May 21, 2013 at 10:03 PM, Michael Segel wrote: > I think what he's missing is to change the configu

Re: Low latency data access Vs High throughput of data

Waoh! I know what latency is and what throughput is , but when someone asks me this question , I was never able to answer it to me satisfaction. Now I can. Thanks a lot! On Wed, May 22, 2013 at 12:21 AM, Jens Scheidtmann < jens.scheidtm...@gmail.com> wrote: > Hi Chris, hi Raj, > > in relational

Re: upgrade from primary + secondary NN to active + standby NN

FWIW, I've seen through a lot of upgrades from 1.x to 2.x and metadata issue's never been a part of it. You should be able to safely upgrade and then to configure HA. Note that you cannot directly upgrade into HA mode yet, you have to upgrade to a regular 1.x to 2.x, and then as a next step, enable

Re: Hadoop Development on cloud in a secure and economical way.

Is it possible to do Hadoop development on cloud in a secure and economical way without worrying about our source being taken away. We would like to have Hadoop and eclipse installed on a vm in cloud and our developers will log into the cloud on a daily basis and work on the cloud. Like this w

Re: Hadoop Development on cloud in a secure and economical way.

Amazon elastic cloud computer. Pay per use Thanks, Rahul On Wed, May 22, 2013 at 11:41 AM, Sai Sai wrote: > > Is it possible to do Hadoop development on cloud in a secure and > economical way without worrying about our source being taken away. We > would like to have Hadoop and eclipse install

Re: Hadoop Development on cloud in a secure and economical way.

Configure private cloud: install VMWare / VirtualBox / KVM on internal server / cluster and levearge either Cloudera Hadoop (free version) or Hortonworks (Yahoo introduced Hortonworks and where Cloudera is exceptional but proprietary Hortonworks requires some configuration and tuning of Hadoop in