Hi German & Thomas,
Seems i found the data that causes the error, but i still don't know the
exactly reason.
I just do a group with pig latin:
domain_device_group = GROUP data_filter BY (custid, domain, level, device);
domain_device = FOREACH domain_device_group {
distinct_ip = D
Is this what you are looking for?
http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/CommandsManual.html#daemonlog
Regards,
*Stanley Shi,*
On Wed, Apr 16, 2014 at 2:06 AM, Ashwin Shankar
wrote:
> Thanks Gordon and Stanley, but this would require us to bounce the process.
>
can do you an "unzip -l myjob.jar" to see if your jar file has the correct
hierarchy?
Regards,
*Stanley Shi,*
On Tue, Apr 15, 2014 at 6:53 PM, laozh...@sina.cn wrote:
> Thank you for your advice . When i user your command , i get the below
> error info .
> $ hadoop jar myjob.jar myjob.MyJob i
Yes, I think you are right.
(2014/04/16 1:20), Manoj Samel wrote:
So, is it correct to say that if one wants to get the latest state of
the Name node, the information from imageviewer and from edits viewer
has to be combined somehow ?
Thanks,
On Tue, Apr 15, 2014 at 7:26 AM, Akira AJISAKA
mai
I think you can use the command 'mvn package -Pnative,dist -DskipTests' in
source code root directory to build the binaries
On Wed, Apr 16, 2014 at 2:31 AM, Justin Mrkva wrote:
> I'm using the guide at
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleCluster.htmlt
It's the same
-- Original --
From: "Shengjun Xin";;
Date: Tue, Apr 15, 2014 04:43 PM
To: "user";
Subject: Re: About "Could not find the main class:
org.apache.hadoop.hdfs.server.namenode.NameNode"
try to use bin/hadoop classpath to check whether the cla
bq. Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build
against old MRv1 mapred APIS)
Which APIs are you talking about, *mapred* or *mapreduce*? In #3, I was
saying about *mapreduce*. If this is the case, you may be in the trouble
unfortunately, because MRv2 has evolved so much
Thanks Zhijie for the explanation.
Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build against
old MRv1 mapred APIS) then how can I compile it since I don't have the source
code i.e. Java files. All I can do with binaries i.e. jar file is execute it.
-RR
Date: Tue, 15 Apr 2014
1. If you have the binaries that were compiled against MRv1 *mapred* libs,
it should just work with MRv2.
2. If you have the source code that refers to MRv1 *mapred* libs, it should
be compilable without code changes. Of course, you're free to change your
code.
3. If you have the binaries that were
Thanks John for your comments,
I believe MRv2 has support for both the old *mapred* APIs and new *mapreduce*
APIs.
I see this way:[1.] One may have binaries i.e. jar file of the M\R program
that used old *mapred* APIsThis will work directly on MRv2(YARN).
[2.] One may have the source code i.e.
Hello All,
For Apache Hadoop 2.x (YARN) installation which *environment variables* are
REALLY needed.
By referring to various blogs I am getting a mix:
HADOOP_COMMON_HOMEHADOOP_CONF_DIRHADOOP_HDFS_HOMEHADOOP_HOMEHADOOP_MAPRED_HOMEHADOOP_PREFIXYARN_HOME
HADOOP_COMMON_HOMEHADOOP_CONF_DIRHADOOP_HDFS_
Hi Shashi,
I am assuming that you are running hadoop 1.x. There is an option to see
the failed tasks on the Job tracker UI. Please replace the jobtracker host
with the actual host and click on the following link and look for the task
failure.
http://[jobtrackerhost]:50030/machines.jsp?type=activ
Hi Saumitra,
It looks like the over replicated blocks root cause is not the issue that
the cluster is experiencing. I can only think of miss configuring the
dfs.data.dir parameter. Can you ensure that each one of the data
directories is using only one partition(mount) and there is no other data
Hello All,
I have configured Apache Hadoop 1.2.0 and set the $HADOOP_HOME env. variable:
I keep getting :Warning: $HADOOP_HOME is deprecated
Solution:(After googling)I replaced HADOOP_HOME with HADOOP_PREFIX and the
warning disappeared.
Does that mean HADOOP_HOME is replaced by HADOOP_PREFIX? If Y
I’m using the guide at
http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleCluster.html
to try to compile the native Hadoop libraries because I’m running a 64 bit OS
and it keeps complaining that the native libraries can’t be found.
After running the third command (mvn
I’m using the guide at
http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleCluster.html
to try to compile the native Hadoop libraries because I’m running a 64 bit OS
and it keeps complaining that the native libraries can’t be found.
After running the third command (mvn
Hi,
Can somebody please help me how to find the task and the datanode in a
large cluster which has failed or which is taking the most time to execute
considering thousands of mappers and reducers are running.
Regards
Shashi
Thanks Gordon and Stanley, but this would require us to bounce the process.
Is there a way to change log levels without bouncing the process ?
On Tue, Apr 15, 2014 at 3:23 AM, Gordon Wang wrote:
> Put the following line in the log4j setting file.
>
> log4j.logger.org.apache.hadoop.yarn.server.
So, is it correct to say that if one wants to get the latest state of the
Name node, the information from imageviewer and from edits viewer has to be
combined somehow ?
Thanks,
On Tue, Apr 15, 2014 at 7:26 AM, Akira AJISAKA
wrote:
> If you want to parse the edits, please use the Offline Edits V
Thanks, let me take a careful look at it.
leiwang...@gmail.com
From: German Florez-Larrahondo
Date: 2014-04-15 23:27
To: user; 'th'
Subject: RE: Re: memoryjava.lang.OutOfMemoryError related with number of
reducer?
Lei
A good explanation of this can be found on the Hadoop The Definitive Guide
Lei
A good explanation of this can be found on the Hadoop The Definitive Guide by
Tom White.
Here is an excerpt that explains a bit the behavior at the reduce side and some
possible tweaks to control it.
https://www.inkling.com/read/hadoop-definitive-guide-tom-white-3rd/chapter-6/shuffle-
Moved to user@hadoop.apache.org.
You can configure the interval by setting
"mapreduce.client.progressmonitor.pollinterval" parameter.
The default value is 1000 ms.
For more details, please see
http://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-defau
Thanks Thomas.
Anohter question. I have no idea what is "Failed to merge in memory". Does
the 'merge' is the shuffle phase in reducer side? Why it is in memory?
Except the two methods(increase reducer number and increase heap size), is
there any other alternatives to fix this issue?
Thank
If you want to parse the edits, please use the Offline Edits Viewer.
http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html
Thanks,
Akira
(2014/04/15 16:41), Mingjiang Shi wrote:
I think you are right because the the offline image viewer only takes
the fsimage
Hi Rahman,
These are few lines from hadoop fsck / -blocks -files -locations
/mnt/hadoop/hive/warehouse/user.db/table1/000255_0 44323326 bytes, 1
block(s): OK
0. blk_-7919979022650423857_446500 len=44323326 repl=3 [ip1:50010,
ip2:50010, ip3:50010]
/mnt/hadoop/hive/warehouse/user.db/table1/000256
When you increase the number of reducers they each have less to work
with provided the data is distributed evenly between them - in this case
about one third of the original work.
It is eessentially the same thing as increasing the heap size - it's
just distributed between more reducers.
/th
On
Pls unsubscribe me. Thx.
在 2013-3-16，上午3:03，kishore raju 写道：
> HI,
>
> We are having an issue where multiple Task Trackers are running out of
> memory. I have collected HeapDump on those TaskTrackers to analyze further.
> They are currently running with 1GB Heap. we are planning to bump it to
I can fix this by changing heap size.
But what confuse me is that when i change the reducer number from 24 to 84,
there's no this error.
Any insight on this?
Thanks
Lei
Failed to merge in memoryjava.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
Thank you for your advice . When i user your command , i get the below error
info .$ hadoop jar myjob.jar myjob.MyJob input outputException in thread "main"
java.lang.ClassNotFoundException: myjob.MyJob
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.r
Put the following line in the log4j setting file.
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager=DEBUG,console
On Tue, Apr 15, 2014 at 8:33 AM, Ashwin Shankar
wrote:
> Hi,
> How do we set log level to debug for lets say only Resource manager
> and not the other hadoop daemons ?
>
>
Trying to use the below command to generate hadoop eclipse plugin, but seem the
directory =/usr/local/hadoop-2.2.0 not correct. I just used the ambari to
installed the hadoop.
$ant jar -Dversion=2.2.0 -Declipse.home=/usr/local/eclipse
-Dhadoop.home=/usr/local/hadoop-2.2.0
error log
BUILD FA
try to use bin/hadoop classpath to check whether the classpath is what you
set
On Tue, Apr 15, 2014 at 4:16 PM, Anacristing <99403...@qq.com> wrote:
> Hi,
>
> I'm trying to setup Hadoop(version 2.2.0) on Windows(32-bit) with
> cygwin(version 1.7.5).
> I export JAVA_HOME=/cygdrive/c/Java/
Hi,
I'm trying to setup Hadoop(version 2.2.0) on Windows(32-bit) with
cygwin(version 1.7.5).
I export JAVA_HOME=/cygdrive/c/Java/jdk1.7.0_51 in hadoop-env.sh
and the classpath is
/home/Administrator/hadoop-2.2.0/etc/hadoop:
/home/Administrator/hadoop-2.2.0/share/hadoop/common/lib/*:
Please use: hadoop jar myjob.jar myjob.MyJob input output
On Tue, Apr 15, 2014 at 3:06 PM, laozh...@sina.cn wrote:
> Hello EveryOne:
> I am new to hadoop,and i am reading Hadoop in action.
> When i tried to run a demo from this book,I got a problem and could not
> find answer from the net.
Please use: hadoop jar myjob.jar myjob.MyJob input output
On Tue, Apr 15, 2014 at 3:06 PM, laozh...@sina.cn wrote:
> Hello EveryOne:
> I am new to hadoop,and i am reading Hadoop in action.
> When i tried to run a demo from this book,I got a problem and could not
> find answer from the net.
I think you are right because the the offline image viewer only takes the
fsimage file as input.
On Tue, Apr 15, 2014 at 9:29 AM, Manoj Samel wrote:
> Hi,
>
> Is it correct to say that the offline image viewer does not accounts for
> any edits that are not yet merged into the fsimage?
>
> Thanks
Hello EveryOne: I am new to hadoop,and i am reading Hadoop in action.When i
tried to run a demo from this book,I got a problem and could not find answer
from the net. Can you help me on this ?
below is the error info :
$ hadoop jar myjob.jar MyJob input outputException in thread "main"
37 matches
Mail list logo
