Hi All,
Could I try other codec besides lzo using hadoop-common TFile property
io.compression.codec.lzo.class? If not, please ignore following information.
I was trying io.compression.codec.lzo.class with value
com.ibm.biginsights.compress.CmxCodec (not the default
Hi Hadoopers,
I got a question about how many blocks does one input split have? It is random
or the number can be configured or fixed(can't be changed)?
Thanks!
Hi,
Check this post:
http://stackoverflow.com/questions/17727468/hadoop-input-split-size-vs-block-size
Regards, D
2014-12-17 15:16 GMT+01:00 Todd bit1...@163.com:
Hi Hadoopers,
I got a question about how many blocks does one input split have? It is
random or the number can be configured
Hello.
FYI.
The way HDFS has been set up, it breaks down very large files into large
blocks(for example, measuring 128MB), and stores three copies of these blocks
ondifferent nodes in the cluster. HDFS has no awareness of the content of
thesefiles. In YARN, when a MapReduce job is started, the
Hi Venkat, you will need sh.exe on your path. It is part of the GnuWin32
toolset.
See BUILDING.txt in the source tree for details.
On Wed, Dec 17, 2014 at 12:30 AM, Venkat Ramakrishnan
venkat.archit...@gmail.com wrote:
Hello,
I am getting an error in 'Project Dist POM', while
generating
Well formulated answer, thanks for sharing!
2014-12-17 17:15 GMT+01:00 mark charts mcha...@yahoo.com:
Hello.
FYI.
The way HDFS has been set up, it breaks down very large files into large
blocks
(for example, measuring 128MB), and stores three copies of these blocks on
different nodes in
Dear All,
I'm pretty new to Hadoop technology and Linux environment hence struggling
even to find solutions for the basic stuff.
For now, Hortonworks Sandbox is working fine for me and i managed to
connect to it thru SSH.
Now i have some csv files in my mac os folders which i want to copy onto
Hi Anil,
you can use the hadoop fs put file or directory and that should add it to
your hdfs
Date: Thu, 18 Dec 2014 09:29:34 +1100
Subject: Copying files to hadoop.
From: anil.jag...@gmail.com
To: user@hadoop.apache.org
Dear All,
I'm pretty new to Hadoop technology and Linux environment hence
Am 17.12.2014 um 23:29 schrieb Anil Jagtap:
Dear All,
I'm pretty new to Hadoop technology and Linux environment hence
struggling even to find solutions for the basic stuff.
For now, Hortonworks Sandbox is working fine for me and i managed to
connect to it thru SSH.
Now i have some csv
Yes i can do that but I have connected from my mac os terminal to linux
using ssh.
Now when I run LS command it shows me list of files folders from Linux
and not from Mac OS.
I have files which I need to put onto Hadoop directly from Mac OS.
So something like below.
From Mac OS Terminal:
Anil,
You have two main options:
1. install the hadoop software on OSX and add the configuration files
appropriate for your sandbox, then do use hdfs dfs –put local remote
2. Setup your sandbox VM to share a directory between OS X and Linux. All
virtual machines that I know of support
what you can do is copy the files to the linux box then use the hadoop fs put.
You can do this like scp /directory/i/want or file.name
username@hostname:/directorytoputfiles/
for example : scp dude.txt dude@main-hadoop:/opt/
Date: Thu, 18 Dec 2014 09:58:43 +1100
Subject: Re: Copying files to
Oh Thanks a lot Wilm.. You understood my problem accurately.. I executed it
and it worked.
I understand i can always copy it to Linux and then put it to hadoop, but i
was just trying to find out if this is possible.
Thanks again.
Rgds, Anil
On Thu, Dec 18, 2014 at 9:56 AM, Wilm Schumacher
Hi,
but I want to point out: my solution works, but is not very smart. I
think Rich has the better answer.
If your file is REALLY huge, then his answer Nr. 1 is the way to go. If
you want to dynamically add and remove files, test this and test that,
than his second answer is a good fit. You just
Anil,
Happy to help!
Cheers,
Rich
Rich Haase | Sr. Software Engineer | Pandora
m 303.887.1146 | rha...@pandora.com
From: Anil Jagtap anil.jag...@gmail.commailto:anil.jag...@gmail.com
Reply-To: user@hadoop.apache.orgmailto:user@hadoop.apache.org
Thanks Mark and Dieter for the reply.
Actually, I got another question in mind. What's the relationship between input
split and mapper task?Is it one one relation or a mapper task can handle more
than one input splits?
If mapper task can only handle one input split, then if there are many
There would be thousands of tasks, but not all fired off at the same time.
The number of parallel tasks is configurable but typically 1 per data node
core.
*...*
On Wed, Dec 17, 2014 at 6:31 PM, bit1...@163.com bit1...@163.com wrote:
Thanks Mark and Dieter for the reply.
Actually, I got
Sure, thanks Mark. That mean, the completed mapper task is not reused to work
on the pending input splits.
bit1...@163.com
From: daemeon reiydelle
Date: 2014-12-18 11:11
To: user
CC: mark charts
Subject: Re: Re: How many blocks does one input split have?
There would be thousands of tasks,
Hi All,
I have configured CDH4 with HA. It was working fine for some time and now I
started seeing this error and namenode had failed over to secondary server.
2014-12-17 08:44:31,847 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
Hi
How can I run c++ programs using opencv libraries in hadoop?
So far I have done MapReduce jobs in Java only..and there we can supply
external jars using command line itself.
And even tried using python language also..to run them we use hadoop
streaming API.
But I am confused how to run C++
Hello,
Can any please explain me how to handle/Resolve the Container crash in YARN
Hadoop.
Thanks
Sajid Syed
Sajid,
Check the logs for your container at
$HADOOP_INSTALL_DIR/logs/userlogs/application_id/container_id. Note that
these will be present on the node where your Application Master is running.
If the container was not able to start, you might get something by printing the
stack trace in
Hi Arpit,
Thanks for responding!
Thanks for pointing that out. I remember reading in one of the
hadoop or pig wikis that unix-like utilities like CYGWIN are not required
anymore for the
windows installation, so I had skipped it initially.
Anyway, I have installed both GNUWIN32 and CYGWIN and
Install git and add the git\bin to the path environment variable.
Most of the UNIX/LINUX commands are available in git\bin
and from Windows command I can use common Unix shell commands ls -l, rm, grep...
On 12/18/14, Venkat Ramakrishnan venkat.archit...@gmail.com wrote:
Hi Arpit,
Thanks for
24 matches
Mail list logo
