Hi ,
1. Make sure Name Node & Job Tracker processes are running or not.
You can verify them using "Jps" command
2. If you use /hadoop/bin/start-dfs.sh it will start only NameNode & Data
Node and not Job Tracker.
To run Job Tracker you need use "start-all.sh" instead of "start-dfs.sh"
3. Verify
Hi Users,
Please help me with some documentation on how to submit job in YARN and
upload files in HDFS. Can I still use the MR1 commands for file uploading
to hadoop fs -put and hadoop jar job.jar input ouput? Because I ran with
errors saying file cannot be uploaded as file cannot found. The direc
Bcc'd dev mailing list.
Hi Andrew,
The memory allocated will always be an integral multiple of minimal allocation
unit which is configured via the property "yarn.scheduler.minimum-allocation-mb
". The default configuration for the value is 1024.
If you change the config to 512, the container m
I've been successful with importing all the leaf-level maven projects as
"Existing Maven Projects" using the eclipse maven plugin. I've also gotten
things to work without the eclipse maven plugin with some combination of
mvn eclipse:eclipse, pointing to the m2repo, and the directory with the top
p
Sandy,
Thanks for all of the tips, I will try this over the weekend. Regarding the
last question, I am still trying to get the source loaded into Eclipse in a
manner that facilitates easier browsing, symbol search, editing, etc. Perhaps
I am just missing some obvious FAQ? This is leading up
there you are again!
thanks!
On Fri, May 31, 2013 at 10:03 AM, Harsh J wrote:
> Hi Julian,
>
> Yes, "dfs" subcommand accepts config overrides via -D. Just do "hadoop
> dfs -Ddfs.replication=X -copyFromLocal …".
>
> On Fri, May 31, 2013 at 10:27 PM, Julian Bui wrote:
> > Hi hadoop users,
> >
>
Hi John,
Here's how I deploy/debug Hadoop locally:
To build and tar Hadoop:
mvn clean package -Pdist -Dtar -DskipTests=true
The tar will be located in the project directory under hadoop-dist/target/.
I untar it into my deploy directory.
I then copy these scripts into the same directory:
had
As others have suggested, you need to call "&op=OPEN" and save the contents
into a buffer which you then write to disk.You can do this with curl by
using '-o /my/local/file.txt' in your curl statement.
-- Adam
On May 31, 2013, at 10:05 AM, Arpit Gupta
wrote:
> There is no api to download
There is no api to download a file. The client you are using to make these
calls will have to read the contents of the file and save it on your local file
system.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On May 31, 2013, at 9:56 AM, Mohammad Mustaqeem <3m.mustaq...@gmail.com> wro
Hi Julian,
Yes, "dfs" subcommand accepts config overrides via -D. Just do "hadoop
dfs -Ddfs.replication=X -copyFromLocal …".
On Fri, May 31, 2013 at 10:27 PM, Julian Bui wrote:
> Hi hadoop users,
>
> I am aware that you can set the replication factor of a file after it's been
> created, but can
This is for reading the content of the file not to download..
I want to download file.
On Fri, May 31, 2013 at 10:23 PM, Arpit Gupta wrote:
> you will have to use the open operation to read the file and save it.
>
>
> http://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-project-dist/hadoop-hdfs/We
Hi hadoop users,
I am aware that you can set the replication factor of a file after it's
been created, but can you do it as you copy files to the HDFS? My
hope/intuition is that if you were able to reduce the replication factor of
a file while copying, the copy time would decrease. I'm finding i
you will have to use the open operation to read the file and save it.
http://hadoop.apache.org/docs/r2.0.3-alpha/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Open_and_Read_a_File
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On May 30, 2013, at 11:44 PM, Mohammad Mustaqeem <3m.mustaq..
Ah, sorry I didn't read the exact problem.
Yes that static call you make to addInputPath goes all the way up to
(inheritance!) FileInputFormat.addInputPath, which just adds input
paths and doesn't automatically imprint itself as the input format
class at the same time.
On Fri, May 31, 2013 at 9:3
Hi
Whats the data per hour or per day u r looking to put into HDFS ?
For dumping source data into HDFS there are again few options
Option 1
===
Have parallel threads dumping raw data into HDFS from your source
Option 2
===
Design how your Objects will look and write code to convert raw i
Dear Harsh,
thanks for your answer. Your post talks about the intermediate and final
result types.
These are already configured in my job as:
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(IntWritable.class);
My problem was input key and value types, though.
Yo
I do not see Raj's response but first, yes you can overwrite data (file) as
many times as you want at the same location in HDFS/Hadoop. Secondly, you
say that the file is small and you indeed want to read it as whole. So, as
I said, then the issue of making sure that the reader task gets the latest
@Raj: so, updating the data and storing them into the same destination
would work?
@Shahab the file is very small, and therefore I am expecting to read it at
once. what would you suggest?
On Fri, May 31, 2013 at 5:30 PM, Shahab Yunus wrote:
> I might not have understood your usecase properly so
I might not have understood your usecase properly so I apologize for that.
But what I think here you need is something outside of Hadoop/HDFS. I am
presuming that you need to read the whole updated file when you are going
to process it with your never-ending job, right? You don't expect to read
it
hadoop assume that you have put the updated file into the input folder.
Raj K Singh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Fri, May 31, 2013 at 8:53 PM, Adamantios Corais <
adamantios.cor...@gmail.com> wrote:
> I am new to
I am new to hadoop so apologize beforehand for my very-fundamental question.
Lets assume that I have a file stored into hadoop that it gets updated once
a day, Also assume that there is a task running at the back end of hadoop
that never stops. How could I reload this file so that hadoop starts
co
Just FYI if you are on linux, KVM and kickstart are really good for this as
well and we have some kickstart Fedora16 hadoop setup scripts I can share
to spin up a cluster of several VMs on the fly with static IPs (that
usually to me is the tricky part with hadoop VM cluster setup - setting up
the V
Hi Sai Sai,
You can take a look at that also: http://goo.gl/iXzae
I just did that yesterday for some other folks I'm working with. Maybe
not the best way, but working like a charm.
JM
2013/5/31 shashwat shriparv :
> Try this
> http://www.youtube.com/watch?v=gIRubPl20oo
> there will be three vid
Hi,
This executable comes with JDK bundle. You can find this in your jdk/bin
directory.
Regards
Jitendra
On Fri, May 31, 2013 at 5:11 PM, shashwat shriparv <
dwivedishash...@gmail.com> wrote:
> C:\Program: command not found??
>
> From where are you running this command is you hadoop is
This is the Hadoop users list. Please ask HBase questions on their
own, vibrant user community at u...@hbase.apache.org for best
responses. I've moved your post there. Please respond back over this
moved address instead of the hadoop lists.
On Fri, May 31, 2013 at 6:00 PM, Ted wrote:
> I tried sc
I tried scouring the API docs as well as googling this and I can't
find a definitive answer.
If I get an HTable instance and I close it, do I have to make sure I'm
finished using the ResultScanner and the Results before I close the
hTable? (i.e. like JDBC connection/resultSets?)
It looks like my
Try this
http://www.youtube.com/watch?v=gIRubPl20oo
there will be three videos 1-3 watch and you can do what you need to do
*Thanks & Regards*
∞
Shashwat Shriparv
On Fri, May 31, 2013 at 5:52 PM, Jitendra Yadav
wrote:
> Hi,
>
> You can create a clone machine through an existing virt
Hi,
You can create a clone machine through an existing virtual machine
in VMware and then run it as a separate virtual machine.
http://www.vmware.com/support/ws55/doc/ws_clone_new_wizard.html
After installing you have to make sure that all the virtual machines are
setup with correct network set
Hi,
Even I have been wanting to know this, I have Oracle VM Virtual box on
windows 7 laptop and inside Oracle VM only one ubuntu instance is running
-how to add multiple virtual machines as Sai Sai has mentioned.
Thanks
Shashidhar
On Fri, May 31, 2013 at 5:23 PM, Sai Sai wrote:
> Just wonderi
Just wondering if anyone has any documentation or references to any articles
how to simulate a multi node cluster setup in 1 laptop with hadoop running on
multiple ubuntu VMs. any help is appreciated.
Thanks
Sai
C:\Program: command not found??
>From where are you running this command is you hadoop is in windows or
linux?
*Thanks & Regards*
∞
Shashwat Shriparv
On Fri, May 31, 2013 at 4:18 PM, 王洪军 wrote:
> $jps
> conform jobtracker is running( namenode and datanode is also needed )
>
$jps
conform jobtracker is running( namenode and datanode is also needed )
2013/5/31 Jagat Singh
> Please run
>
> $ jps
>
> This command will show all running Hadoop daemons and then you can find
> whats wrong :)
>
>
>
>
> On Fri, May 31, 2013 at 8:25 PM, Mohammad Tariq wrote:
>
>> Hello sumit,
Please run
$ jps
This command will show all running Hadoop daemons and then you can find
whats wrong :)
On Fri, May 31, 2013 at 8:25 PM, Mohammad Tariq wrote:
> Hello sumit,
>
> Make sure all the Hadoop daemons are running .
>
> Warm Regards,
> Tariq
> cloudfront.blogspot.com
>
>
> On
Hello sumit,
Make sure all the Hadoop daemons are running .
Warm Regards,
Tariq
cloudfront.blogspot.com
On Fri, May 31, 2013 at 3:07 PM, sumit piparsania wrote:
> Hi,
>
> I am new to hadoop. I am facing some issues while executing the below
> command.
> Kindly help me resolving this iss
Hi,
I am new to hadoop. I am facing some issues while executing the below command.
Kindly help me resolving this issue.
command: bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'
Error:
bin/hadoop: line 320: C:\Program: command not found
13/05/31 12:59:58 INFO ipc.Client: Ret
Ingesting the data in HDFS is slow ,Because it need a jvm process. But if
you don't use hdfs, you can't benifit from its features. Without hdfs,the
big data will not be splited and distributed; I think the initial time of
jvm is affordable if data is big, and hadoop is not good choice if the da
Then why not simply run with Write Replication Factor set to 1?
On Fri, May 31, 2013 at 12:54 PM, Agarwal, Nikhil
wrote:
> Hi,
>
>
>
> Thank you for your reply. One simple answer can be to reduce the time taken
> for ingesting the data in HDFS.
>
>
>
> Regards,
>
> Nikhil
>
>
>
> From: Sanjay Sub
Hi,
Thank you for your reply. One simple answer can be to reduce the time taken for
ingesting the data in HDFS.
Regards,
Nikhil
From: Sanjay Subramanian [mailto:sanjay.subraman...@wizecommerce.com]
Sent: Friday, May 31, 2013 12:50 PM
To:
Cc: user@hadoop.apache.org
Subject: Re: MapReduce on Loc
Basic question. Why would u want to do that ? Also I think the Map R Hadoop
distribution has an NFS mountable HDFS
Sanjay
Sent from my iPhone
On May 30, 2013, at 11:37 PM, "Agarwal, Nikhil"
mailto:nikhil.agar...@netapp.com>> wrote:
Hi,
Is it possible to run MapReduce on multiple nodes using L
yeah , I meant nfs mount.
thanks,
Rahul
On Fri, May 31, 2013 at 12:42 PM, Agarwal, Nikhil wrote:
> Hi Rahul,
>
> ** **
>
> Can you please explain what do you mean by “filer directory mounted to
> all the DN” ? Do you mean a NFS-mount? If yes then I want to avoid
> NFS-mount. With NFS-moun
Hi Rahul,
Can you please explain what do you mean by “filer directory mounted to all the
DN” ? Do you mean a NFS-mount? If yes then I want to avoid NFS-mount. With
NFS-mount it is possible to do it.
Thanks & Regards,
Nikhil
From: Rahul Bhattacharjee [mailto:rahul.rec@gmail.com]
Sent: Frida
Just a hunch. Can have a filer directory mounted to all the DN and then
file:/// should be usuable in a distributed fashion. (Just a guess)
Thanks,
Rahul
On Fri, May 31, 2013 at 12:07 PM, Agarwal, Nikhil wrote:
> Hi,
>
> ** **
>
> Is it possible to run MapReduce on *multiple nodes* using
42 matches
Mail list logo
