RE: launch aggregatewordcount and sudoku in Yarn
Hi, You can run aggregatewordcount Job using the below command, Command : bin/hadoop jar hadoop-*-examples.jar aggregatewordcount in-dir out-dir numOfReducers inputformat inputformat - It uses default as seqfileinputformat, we can change it to textinputformat. If we pass other than textinputformat it will use seqfileinputformat. Ex : bin/hadoop jar hadoop-*-examples.jar aggregatewordcount /input-data /out-dir 5 textinputformat You can run the Sudoku Job using the below command, Command : bin/hadoop jar hadoop-*-examples.jar sudoku puzzle-to-solve puzzle-to-solve - You need to give the puzzle as input to this sudoku job. You can have the puzzle in a file which is present in local location and same name you can pass it. Ex : bin/hadoop jar hadoop-*-examples.jar sudoku puzzle1.dta You can place the attached puzzle1.dta file in local location and same thing you can pass as argument to this. Thanks Devaraj K From: Pedro Sá da Costa [mailto:psdc1...@gmail.com] Sent: 21 June 2013 22:24 To: mapreduce-user Subject: launch aggregatewordcount and sudoku in Yarn How I run an aggregatewordcount and sudoku in Yarn? Do I need any input files, more exactly in Sudoku? -- Best regards, puzzle1.dta Description: puzzle1.dta
RE: How run Aggregator wordcount?
It doesn't accept multiple folders as input. You can have multiple files in a directory and same you can give as input. Thanks Devaraj K From: Pedro Sá da Costa [mailto:psdc1...@gmail.com] Sent: 22 June 2013 16:25 To: mapreduce-user Subject: How run Aggregator wordcount? Aggregator wordcount accept multiple folders as input? e.g. bin/hadoop jar hadoop-*-examples.jar aggregatewordcount inputfolder1 inputfolder2 inputfolder3 outfolder1 -- Best regards,
Re: MapReduce job not running - i think i keep all correct configuration.
Can you paste some error logs here? you can find it on the JT or TT. and tell us the hadoop version. On Sun, Jun 23, 2013 at 9:20 PM, Pavan Kumar Polineni smartsunny...@gmail.com wrote: Hi all, first i have a machine with all the demons are running on it. after that i added two data nodes. In this case MR job working fine. Now i changed the first machine to just namenode by stopping all the demons except NN demon. and changed i data node to (SNN.JT,DN,TT) and all are working. i keep the other data node like that only. I changed the configurations to link up the NN and JT. From here when i tried to run MR job this is not running .. Please help Me. Thanks -- Pavan Kumar Polineni
Re: MapReduce job not running - i think i keep all correct configuration.
Hi all, i solved it. due to some cluster id prob . Thanks for support On Mon, Jun 24, 2013 at 11:39 AM, Azuryy Yu azury...@gmail.com wrote: Can you paste some error logs here? you can find it on the JT or TT. and tell us the hadoop version. On Sun, Jun 23, 2013 at 9:20 PM, Pavan Kumar Polineni smartsunny...@gmail.com wrote: Hi all, first i have a machine with all the demons are running on it. after that i added two data nodes. In this case MR job working fine. Now i changed the first machine to just namenode by stopping all the demons except NN demon. and changed i data node to (SNN.JT,DN,TT) and all are working. i keep the other data node like that only. I changed the configurations to link up the NN and JT. From here when i tried to run MR job this is not running .. Please help Me. Thanks -- Pavan Kumar Polineni -- Pavan Kumar Polineni
Re: how to turn on NativeIO.posixFadviseIfPossible
Hi Leo,
Following your instruction on moving the sections in configure.ac, I am
able to re-compile the code base and I can find that the objdump returns
the correct fadvise symbols as you described in your earlier email. I then
ran the TestNativeIO.java class and I believe that the Fadvise test case
gets passed now.
Thank you very much for the help!
Regards,
Jun
On Sat, Jun 22, 2013 at 4:42 PM, Leo Leung lle...@ddn.com wrote:
Hi Jun,
** **
Try moving the section between:
dnl Check for headers needed by the native Group….
[…include everything in between inclusive …]
AC_FUNC_STRERROR_R
to higher place (insert before)
** **
# Checks for libraries.
dnl Check for ‘-ldl’
Hope this helps J
** **
** **
*From:* Jun Li [mailto:jltz922...@gmail.com]
*Sent:* Saturday, June 22, 2013 3:26 AM
*To:* user@hadoop.apache.org
*Subject:* Re: how to turn on NativeIO.posixFadviseIfPossible
** **
Hi Leo,
Thanks for the detailed instruction. I started from the clean downloaded
hadoop1.1.12 copy and then I issue the command: ant -Dcompile.native=true.
And with the configure.ac that comes with the downloaded package, without
any changes, the following is the compilation output:
[exec] checking fcntl.h usability... yes
[exec] checking fcntl.h presence... yes
[exec] checking for fcntl.h... yes
[exec] checking for stdlib.h... (cached) yes
[exec] checking for string.h... (cached) yes
[exec] checking for unistd.h... (cached) yes
[exec] checking for fcntl.h... (cached) yes
[exec] checking for posix_fadvise... no
[exec] checking for fcntl.h... (cached) yes
[exec] checking for sync_file_range... no
[exec] checking for an ANSI C-conforming const... yes
[exec] checking for memset... no
[exec] checking whether strerror_r is declared... yes
[exec] checking for strerror_r... no
I have difficulty to follow the + and - relate patching sequence that
you stated in your email earlier. I have attached the configure.ac file
that I copied out of the hadoop1.1.12/src/native directory. Could you help
me to construct the configure.ac with the correct patching that you
described, using the one that I attached in this email, and send back to
me? Then I will use the patched one to build the native library.
Regards,
Jun
** **
On Fri, Jun 21, 2013 at 12:35 PM, Leo Leung lle...@ddn.com wrote:
Hi Jun:
Looks like it is not in the apache hadoop 1.x release binaries, this
probably will have to be answered by the apache hadoop 1.x release team***
*
To build the native libs:
1) Get the source code and install the dependencies to compile hadoop
2) You’ll have to setup the env for it, such as JAVA_HOME. Gcc
installed etc.
3) Got to hadoop-source dir ant [veryclean]
compile-core-native compile.out
Verify with
[branch-1]$ objdump -Tt ./build/native/Linux-amd64-64/lib/libhadoop.so |
grep fadv
56a0 g F .text 00a3
Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
F *UND*
posix_fadvise@@GLIBC_2.2.5
DF *UND* GLIBC_2.2.5 posix_fadvise
56a0 gDF .text 00a3 Base
Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
If you get an error about “make distclean” error
Just cd in src/native and run make distclean (to cleanup )
Example of ant output that shows checking for fadvise
[exec] checking for fcntl.h... yes
[exec] checking for stdlib.h... (cached) yes
[exec] checking for string.h... (cached) yes
[exec] checking for unistd.h... (cached) yes
[exec] checking for fcntl.h... (cached) yes
[exec] checking for posix_fadvise... yes
[exec] checking for fcntl.h... (cached) yes
[exec] checking for sync_file_range... yes
If you see “no” for posix_fadvise or “sync_file_range”
And you are sure you have posix_fdavise support (RHEL 6.x or CentOS 6.x)**
**
you may need to move the section on src/native/configure.ac
Try to modify the configure.ac
--- src/native/configure.ac (revision 1495498)
+++ src/native/configure.ac (working copy)
@@ -47,6 +47,21 @@
AC_PROG_CC
AC_PROG_LIBTOOL
+dnl check for posix_fadvise
+AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(posix_fadvise)])
+
+dnl check for sync_file_range
+AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(sync_file_range)])
+
+# Checks for typedefs, structures, and compiler characteristics.
+AC_C_CONST
+
+# Checks for library functions.
+AC_CHECK_FUNCS([memset])
+
+# Check for nonstandard STRERROR_R
RE: Job end notification does not always work (Hadoop 2.x)
Even if we handle all the failure cases in AM for Job End Notification, we may
miss cases like abrupt kill of AM when it is in last retry. If we choose NM to
give the notification, again RM needs to identify which NM should give the
end-notification as we don't have any direct protocol between AM and NM.
I feel it would be better to move End-Notification responsibility to RM as Yarn
Service because it ensures 100% notification and also useful for other types of
applications as well.
Thanks
Devaraj K
From: Ravi Prakash [mailto:ravi...@ymail.com]
Sent: 23 June 2013 19:01
To: user@hadoop.apache.org
Subject: Re: Job end notification does not always work (Hadoop 2.x)
Hi Alejandro,
Thanks for your reply! I was thinking more along the lines Prashant suggested
i.e. a failure during init() should still trigger an attempt to notify (by the
AM). But now that you mention it, maybe we would be better of including this as
a YARN feature after all (specially with all the new AMs being written). We
could let the NM of the AM handle the notification burden, so that the RM
doesn't get unduly taxed. Thoughts?
Thanks
Ravi
From: Alejandro Abdelnur t...@cloudera.commailto:t...@cloudera.com
To: common-u...@hadoop.apache.orgmailto:common-u...@hadoop.apache.org
user@hadoop.apache.orgmailto:user@hadoop.apache.org
Sent: Saturday, June 22, 2013 7:37 PM
Subject: Re: Job end notification does not always work (Hadoop 2.x)
If the AM fails before doing the job end notification, at any stage of the
execution for whatever reason, the job end notification will never be deliver.
There is not way to fix this unless the notification is done by a Yarn service.
The 2 'candidate' services for doing this would be the RM and the HS. The job
notification URL is in the job conf. The RM never sees the job conf, that rules
out the RM out unless we add, at AM registration time the possibility to
specify a callback URL. The HS has access to the job conf, but the HS is
currently a 'passive' service.
thx
On Sat, Jun 22, 2013 at 3:48 PM, Arun C Murthy
a...@hortonworks.commailto:a...@hortonworks.com wrote:
Prashanth,
Please file a jira.
One thing to be aware of - AMs get restarted a certain number of times for
fault-tolerance - which means we can't just assume that failure of a single AM
is equivalent to failure of the job.
Only the ResourceManager is in the appropriate position to judge failure of AM
v/s failure-of-job.
hth,
Arun
On Jun 22, 2013, at 2:44 PM, Prashant Kommireddi
prash1...@gmail.commailto:prash1...@gmail.com wrote:
Thanks Ravi.
Well, in this case its a no-effort :) A failure of AM init should be considered
as failure of the job? I looked at the code and best-effort makes sense with
respect to retry logic etc. You make a good point that there would be no
notification in case AM OOMs, but I do feel AM init failure should send a
notification by other means.
On Sat, Jun 22, 2013 at 2:38 PM, Ravi Prakash
ravi...@ymail.commailto:ravi...@ymail.com wrote:
Hi Prashant,
I would tend to agree with you. Although job-end notification is only a
best-effort mechanism (i.e. we cannot always guarantee notification for
example when the AM OOMs), I agree with you that we can do more. If you feel
strongly about this, please create a JIRA and possibly upload a patch.
Thanks
Ravi
From: Prashant Kommireddi prash1...@gmail.commailto:prash1...@gmail.com
To: user@hadoop.apache.orgmailto:user@hadoop.apache.org
user@hadoop.apache.orgmailto:user@hadoop.apache.org
Sent: Thursday, June 20, 2013 9:45 PM
Subject: Job end notification does not always work (Hadoop 2.x)
Hello,
I came across an issue that occurs with the job notification callbacks in MR2.
It works fine if the Application master has started, but does not send a
callback if the initializing of AM fails.
Here is the code from MRAppMaster.java
.
...
// set job classloader if configured
MRApps.setJobClassLoader(conf);
initAndStartAppMaster(appMaster, conf, jobUserName);
} catch (Throwable t) {
LOG.fatal(Error starting MRAppMaster, t);
System.exit(1);
}
}
protected static void initAndStartAppMaster(final MRAppMaster appMaster,
final YarnConfiguration conf, String jobUserName) throws IOException,
InterruptedException {
UserGroupInformation.setConfiguration(conf);
UserGroupInformation appMasterUgi = UserGroupInformation
.createRemoteUser(jobUserName);
appMasterUgi.doAs(new PrivilegedExceptionActionObject() {
@Override
public Object run() throws Exception {
appMaster.init(conf);
appMaster.start();
if(appMaster.errorHappenedShutDown) {
throw new IOException(Was asked to shut down.);
}
return null;
}
});
}
appMaster.init(conf) does not dispatch JobFinishEventHandler which is
responsible for sending a HTTP callback
Re: Requesting containers on a specific host
Yes Arun, the one I am giving is same as the one I see in RM's log also. Thanks, Kishore On Fri, Jun 21, 2013 at 7:25 PM, Arun C Murthy a...@hortonworks.com wrote: Check if the hostname you are setting is the same in the RM logs… On Jun 21, 2013, at 2:15 AM, Krishna Kishore Bonagiri write2kish...@gmail.com wrote: Hi, I am trying to get container on a specific host, using setHostName(0 call on ResourceRequest, but could not get allocated anything forever, which just works fine when I change the node name to *. I am working on a single node cluster, and I am giving the name of the single node I have in my cluster. Is there any specific format that I need to give for setHostName(), why is it not working... Thanks, Kishore -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: Job end notification does not always work (Hadoop 2.x)
if we ought to do this in a yarn service it
should be the RM or the HS. the RM is, IMO, the natural fit. the HS, would be a
good choice if we are concerned about the extra work this would cause in the
RM. the problem with the current HS is that it is MR specific, we should
generalize it for diff AM types.
thx
Alejandro
(phone typing)
On Jun 23, 2013, at 23:28, Devaraj k devara...@huawei.com wrote:
Even if we handle all the failure cases in AM for Job End Notification, we
may miss cases like abrupt kill of AM when it is in last retry. If we choose
NM to give the notification, again RM needs to identify which NM should give
the end-notification as we don't have any direct protocol between AM and NM.
I feel it would be better to move End-Notification responsibility to RM as
Yarn Service because it ensures 100% notification and also useful for other
types of applications as well.
Thanks
Devaraj K
From: Ravi Prakash [mailto:ravi...@ymail.com]
Sent: 23 June 2013 19:01
To: user@hadoop.apache.org
Subject: Re: Job end notification does not always work (Hadoop 2.x)
Hi Alejandro,
Thanks for your reply! I was thinking more along the lines Prashant suggested
i.e. a failure during init() should still trigger an attempt to notify (by
the AM). But now that you mention it, maybe we would be better of including
this as a YARN feature after all (specially with all the new AMs being
written). We could let the NM of the AM handle the notification burden, so
that the RM doesn't get unduly taxed. Thoughts?
Thanks
Ravi
From: Alejandro Abdelnur t...@cloudera.com
To: common-u...@hadoop.apache.org user@hadoop.apache.org
Sent: Saturday, June 22, 2013 7:37 PM
Subject: Re: Job end notification does not always work (Hadoop 2.x)
If the AM fails before doing the job end notification, at any stage of the
execution for whatever reason, the job end notification will never be
deliver. There is not way to fix this unless the notification is done by a
Yarn service. The 2 'candidate' services for doing this would be the RM and
the HS. The job notification URL is in the job conf. The RM never sees the
job conf, that rules out the RM out unless we add, at AM registration time
the possibility to specify a callback URL. The HS has access to the job conf,
but the HS is currently a 'passive' service.
thx
On Sat, Jun 22, 2013 at 3:48 PM, Arun C Murthy a...@hortonworks.com wrote:
Prashanth,
Please file a jira.
One thing to be aware of - AMs get restarted a certain number of times for
fault-tolerance - which means we can't just assume that failure of a single
AM is equivalent to failure of the job.
Only the ResourceManager is in the appropriate position to judge failure of
AM v/s failure-of-job.
hth,
Arun
On Jun 22, 2013, at 2:44 PM, Prashant Kommireddi prash1...@gmail.com wrote:
Thanks Ravi.
Well, in this case its a no-effort :) A failure of AM init should be
considered as failure of the job? I looked at the code and best-effort makes
sense with respect to retry logic etc. You make a good point that there would
be no notification in case AM OOMs, but I do feel AM init failure should send
a notification by other means.
On Sat, Jun 22, 2013 at 2:38 PM, Ravi Prakash ravi...@ymail.com wrote:
Hi Prashant,
I would tend to agree with you. Although job-end notification is only a
best-effort mechanism (i.e. we cannot always guarantee notification for
example when the AM OOMs), I agree with you that we can do more. If you feel
strongly about this, please create a JIRA and possibly upload a patch.
Thanks
Ravi
From: Prashant Kommireddi prash1...@gmail.com
To: user@hadoop.apache.org user@hadoop.apache.org
Sent: Thursday, June 20, 2013 9:45 PM
Subject: Job end notification does not always work (Hadoop 2.x)
Hello,
I came across an issue that occurs with the job notification callbacks in
MR2. It works fine if the Application master has started, but does not send a
callback if the initializing of AM fails.
Here is the code from MRAppMaster.java
.
...
// set job classloader if configured
MRApps.setJobClassLoader(conf);
initAndStartAppMaster(appMaster, conf, jobUserName);
} catch (Throwable t) {
LOG.fatal(Error starting MRAppMaster, t);
System.exit(1);
}
}
protected static void initAndStartAppMaster(final MRAppMaster appMaster,
final YarnConfiguration conf, String jobUserName) throws IOException,
InterruptedException {
UserGroupInformation.setConfiguration(conf);
UserGroupInformation appMasterUgi = UserGroupInformation
.createRemoteUser(jobUserName);
appMasterUgi.doAs(new PrivilegedExceptionActionObject() {
@Override
public Object run() throws Exception {
appMaster.init(conf);
appMaster.start();
if(appMaster.errorHappenedShutDown) {
Re: Compile Just a Subproject
Hi Omkar, That was a typo, good catch. The path to Client.java should be C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn- applications-distributedshell /src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java Running as you suggest does build the jar in target with an updated timestamp. I just need to copy it into C:\l\hadoop-dist\target\hadoop-3.0.0-SNAPSHOT\share\hadoop\yarn and override the version packaged there, and then things run as expected. Thanks for your help! Curtis On Fri, Jun 21, 2013 at 7:49 PM, Omkar Joshi ojo...@hortonworks.com wrote: Hi Curtis...I see a mismatch in paths which you have mentioned..are they same and was a typo or a mistake? ...modify logging code in *C:/l/hadoop-yarn-applications-** distributedshell/src/main/java/org/apache/hadoop/yarn/ applications/distributedshell/Client.java* * * and * C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell *mvn clean install -DskipTests C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshellhadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command whoami * * looks like you also have another directory named * hadoop-yarn-applications-**distributedshell *copied inside c:\1\... to make it more simple...go to *C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\ directory and clean everything and then run the mvn clean install command from **C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell directory.* * * * * *Let me know if you can now see the jar file (with updated time stamp) or not in target directory.* Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Jun 21, 2013 at 3:12 PM, Curtis Ullerich curtuller...@gmail.comwrote: I've executed the commands as you've said, and the jar that is run is unchanged. Here's exactly what I did. C:\l\ start-dfs C:\l\ start-yarn C:\l\ cd hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshellhadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command whoami ...output... ...modify logging code in C:/l/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshellmvn clean install -DskipTests C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshellhadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command whoami ...identical output to before... Do you see anything wrong with what I've done? Thanks, Curtis On Thu, Jun 20, 2013 at 7:17 PM, Omkar Joshi ojo...@hortonworks.comwrote: Hi Curtis, where are you picking your jar file from? once you run above command you will see the updated jar file in /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar I hope you are not using below jar file /hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar What chris has said is right. once you have taken latest code, you should follow below steps 1) mvn clean install -DskipTests (clean to remove previously generated code) 2) now say you are updating distributed shell client code. then go to /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/ and then run mvn clean install and use jar from target sub folder. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Thu, Jun 20, 2013 at 11:47 AM, Curtis Ullerich curtuller...@gmail.com wrote: Hi Chris, I really appreciate the response. What you described is what I initially tried. The changes never seem to take effect though. Here's what I've done (this is Windows): cd %hadoop_install_dir% mvn clean package -DskipTests mvn install -DskipTests --- modify the code in distributed shell's Client.java --- cd hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell mvn clean mvn package -DskipTests mvn install -DskipTests Then I run the jar, just as before. I've just been changing log statements to see if the changes worked. They
RE: how to turn on NativeIO.posixFadviseIfPossible
You're welcome. Have fun with hadoop! Regards, Leo From: Jun Li [mailto:jltz922...@gmail.com] Sent: Sunday, June 23, 2013 11:24 PM To: user@hadoop.apache.org Subject: Re: how to turn on NativeIO.posixFadviseIfPossible Hi Leo, Following your instruction on moving the sections in configure.achttp://configure.ac, I am able to re-compile the code base and I can find that the objdump returns the correct fadvise symbols as you described in your earlier email. I then ran the TestNativeIO.java class and I believe that the Fadvise test case gets passed now. Thank you very much for the help! Regards, Jun On Sat, Jun 22, 2013 at 4:42 PM, Leo Leung lle...@ddn.commailto:lle...@ddn.com wrote: Hi Jun, Try moving the section between: dnl Check for headers needed by the native Group [...include everything in between inclusive ...] AC_FUNC_STRERROR_R to higher place (insert before) # Checks for libraries. dnl Check for '-ldl' Hope this helps :) From: Jun Li [mailto:jltz922...@gmail.commailto:jltz922...@gmail.com] Sent: Saturday, June 22, 2013 3:26 AM To: user@hadoop.apache.orgmailto:user@hadoop.apache.org Subject: Re: how to turn on NativeIO.posixFadviseIfPossible Hi Leo, Thanks for the detailed instruction. I started from the clean downloaded hadoop1.1.12 copy and then I issue the command: ant -Dcompile.native=true. And with the configure.achttp://configure.ac that comes with the downloaded package, without any changes, the following is the compilation output: [exec] checking fcntl.h usability... yes [exec] checking fcntl.h presence... yes [exec] checking for fcntl.h... yes [exec] checking for stdlib.h... (cached) yes [exec] checking for string.h... (cached) yes [exec] checking for unistd.h... (cached) yes [exec] checking for fcntl.h... (cached) yes [exec] checking for posix_fadvise... no [exec] checking for fcntl.h... (cached) yes [exec] checking for sync_file_range... no [exec] checking for an ANSI C-conforming const... yes [exec] checking for memset... no [exec] checking whether strerror_r is declared... yes [exec] checking for strerror_r... no I have difficulty to follow the + and - relate patching sequence that you stated in your email earlier. I have attached the configure.achttp://configure.ac file that I copied out of the hadoop1.1.12/src/native directory. Could you help me to construct the configure.achttp://configure.ac with the correct patching that you described, using the one that I attached in this email, and send back to me? Then I will use the patched one to build the native library. Regards, Jun On Fri, Jun 21, 2013 at 12:35 PM, Leo Leung lle...@ddn.commailto:lle...@ddn.com wrote: Hi Jun: Looks like it is not in the apache hadoop 1.x release binaries, this probably will have to be answered by the apache hadoop 1.x release team To build the native libs: 1) Get the source code and install the dependencies to compile hadoop 2) You'll have to setup the env for it, such as JAVA_HOME. Gcc installed etc. 3) Got to hadoop-source dir ant [veryclean] compile-core-native compile.out Verify with [branch-1]$ objdump -Tt ./build/native/Linux-amd64-64/lib/libhadoop.so | grep fadv 56a0 g F .text 00a3 Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise F *UND* posix_fadvise@@GLIBC_2.2.5 DF *UND* GLIBC_2.2.5 posix_fadvise 56a0 gDF .text 00a3 Base Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise If you get an error about make distclean error Just cd in src/native and run make distclean (to cleanup ) Example of ant output that shows checking for fadvise [exec] checking for fcntl.h... yes [exec] checking for stdlib.h... (cached) yes [exec] checking for string.h... (cached) yes [exec] checking for unistd.h... (cached) yes [exec] checking for fcntl.h... (cached) yes [exec] checking for posix_fadvise... yes [exec] checking for fcntl.h... (cached) yes [exec] checking for sync_file_range... yes If you see no for posix_fadvise or sync_file_range And you are sure you have posix_fdavise support (RHEL 6.x or CentOS 6.x) you may need to move the section on src/native/configure.achttp://configure.ac Try to modify the configure.achttp://configure.ac --- src/native/configure.achttp://configure.ac (revision 1495498) +++ src/native/configure.achttp://configure.ac (working copy) @@ -47,6 +47,21 @@ AC_PROG_CC AC_PROG_LIBTOOL +dnl check for posix_fadvise +AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(posix_fadvise)]) + +dnl check for sync_file_range +AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(sync_file_range)]) + +# Checks for typedefs, structures, and compiler characteristics. +AC_C_CONST + +# Checks for library functions. +AC_CHECK_FUNCS([memset]) + +# Check
Piping to HDFS (from Linux or HDFS)
Hi guys While I was trying to get some test data and configurations done quickly I realized one can do this and I think its super cool Processing existing file on Linux/HDFS and Piping it directly to hdfs source = Linux dest=HDFS == File = sanjay.conf.template We want to replace one line in the file -99-99 1947-08-15 DATE_STR=-99-99 cat sanjay.conf.template | sed 's/-99-99/1947-08-15/g' | hdfs dfs -put - /user/sanjay/sanjay.conf source = HDFS dest=HDFS == hdfs dfs -cat /user/nextag/sanjay.conf.template | sed 's/-99-99/1947-08-15/g' | hdfs dfs -put - /user/sanjay/1947-08-15/nextag.conf Thanks sanjay CONFIDENTIALITY NOTICE == This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
could only be replicated to 0 nodes instead of minReplication exception during job execution
Hello,
I am using YARN. I get some exceptions at my namenode and datanode. They
are thrown when my Reduce progress gets 67%. Then, reduce phase is
restarted from 0% several times, but always restarts at this point. Can
someone tell me what I should do? Many thanks!
Namenode log:
2013-06-24 19:08:50,345 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.2.190:50010 is added to
blk_654446797771285606_5062{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.224.2.190:50010|RBW]]} size 0
2013-06-24 19:08:50,349 WARN
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy:
Not able to place enough replicas, still in need of 1 to reach 1
For more information, please enable DEBUG log level on
org.apache.commons.logging.impl.Log4JLogger
2013-06-24 19:08:50,350 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:ubuntu (auth:SIMPLE)
cause:java.io.IOException: File
/output/_temporary/1/_temporary/attempt_1372090853102_0001_r_02_0/part-2
could only be replicated to 0 nodes instead of minReplication (=1).
There are 2 datanode(s) running and no node(s) are excluded in this
operation.
2013-06-24 19:08:50,353 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 9000, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
10.224.2.190:49375: error: java.io.IOException: File
/output/_temporary/1/_temporary/attempt_1372090853102_0001_r_02_0/part-2
could only be replicated to 0 nodes instead of minReplication (=1).
There are 2 datanode(s) running and no node(s) are excluded in this
operation.
java.io.IOException: File
/output/_temporary/1/_temporary/attempt_1372090853102_0001_r_02_0/part-2
could only be replicated to 0 nodes instead of minReplication (=1).
There are 2 datanode(s) running and no node(s) are excluded in this
operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1339)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2155)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:491)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:351)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40744)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
2013-06-24 19:08:50,413 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.2.190:50010 is added to
blk_8924314838535676494_5063{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.224.2.190:50010|RBW]]} size 0
2013-06-24 19:08:50,418 WARN
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy:
Not able to place enough replicas, still in need of 1 to reach 1
For more information, please enable DEBUG log level on
org.apache.commons.logging.impl.Log4JLogger
Datanode log:
2013-06-24 19:25:54,695 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-1724882733-10.10.79.145-1372090400593:blk_-2417373821601940925_6022,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2013-06-24 19:25:54,699 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving
BP-1724882733-10.10.79.145-1372090400593:blk_3177955398059619584_6033 src: /
10.35.99.108:59710 dest: /10.35.99.108:50010
2013-06-24 19:25:56,473 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for
BP-1724882733-10.10.79.145-1372090400593:blk_8751401862589207807_6026
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)
at sun.nio.ch.IOUtil.read(IOUtil.java:224)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:159)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
at
Re: could only be replicated to 0 nodes instead of minReplication exception during job execution
Hi,
I see there are 2 datanodes and for some reason namenode is not able to
create even single replica for requested blocks. are you sure the system on
which these datanodes are running have sufficient disk space? Do you see
any other errors in datanode/namenode logs?
What must be happening is as file creation in hdfs is failing it is marking
that reduce attempt as failure and restarting it. Keep checking namenode
state when it reaches 67%.
Thanks,
Omkar Joshi
*Hortonworks Inc.* http://www.hortonworks.com
On Mon, Jun 24, 2013 at 3:01 PM, Yuzhang Han yuzhanghan1...@gmail.comwrote:
Hello,
I am using YARN. I get some exceptions at my namenode and datanode. They
are thrown when my Reduce progress gets 67%. Then, reduce phase is
restarted from 0% several times, but always restarts at this point. Can
someone tell me what I should do? Many thanks!
Namenode log:
2013-06-24 19:08:50,345 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.2.190:50010 is added to
blk_654446797771285606_5062{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.224.2.190:50010|RBW]]} size 0
2013-06-24 19:08:50,349 WARN
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not able
to place enough replicas, still in need of 1 to reach 1
For more information, please enable DEBUG log level on
org.apache.commons.logging.impl.Log4JLogger
2013-06-24 19:08:50,350 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:ubuntu (auth:SIMPLE) cause:java.io.IOException: File
/output/_temporary/1/_temporary/attempt_1372090853102_0001_r_02_0/part-2
could only be replicated to 0 nodes instead of minReplication (=1). There
are 2 datanode(s) running and no node(s) are excluded in this operation.
2013-06-24 19:08:50,353 INFO org.apache.hadoop.ipc.Server: IPC Server handler
1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
10.224.2.190:49375: error: java.io.IOException: File
/output/_temporary/1/_temporary/attempt_1372090853102_0001_r_02_0/part-2
could only be replicated to 0 nodes instead of minReplication (=1). There
are 2 datanode(s) running and no node(s) are excluded in this operation.
java.io.IOException: File
/output/_temporary/1/_temporary/attempt_1372090853102_0001_r_02_0/part-2
could only be replicated to 0 nodes instead of minReplication (=1). There
are 2 datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1339)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2155)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:491)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:351)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40744)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
2013-06-24 19:08:50,413 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.2.190:50010 is added to
blk_8924314838535676494_5063{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.224.2.190:50010|RBW]]} size 0
2013-06-24 19:08:50,418 WARN
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not able
to place enough replicas, still in need of 1 to reach 1
For more information, please enable DEBUG log level on
org.apache.commons.logging.impl.Log4JLogger
Datanode log:
2013-06-24 19:25:54,695 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-1724882733-10.10.79.145-1372090400593:blk_-2417373821601940925_6022,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2013-06-24 19:25:54,699 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving
BP-1724882733-10.10.79.145-1372090400593:blk_3177955398059619584_6033 src: /
10.35.99.108:59710 dest: /10.35.99.108:50010
2013-06-24 19:25:56,473 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for
BP-1724882733-10.10.79.145-1372090400593:blk_8751401862589207807_6026
java.io.IOException: Connection
which hadoop version i can choose in production env?
hadoop 1 vs hadoop 2 ,and apache community version vs cloudrea version,anyone can help?
Re: which hadoop version i can choose in production env?
I advice community version of Hadoop-1.1.2, which is a stable release, Hadoop2 hasn't stable release currently, even if all alpha release was extensive tested. but from me, I think HDFS2 is stable now.(no?), MR1 is also stable, but Yarn still need extensive tests(at least I think so), so our prod cluster using HDFS2 and MR1 now. On Tue, Jun 25, 2013 at 10:11 AM, ch huang justlo...@gmail.com wrote: hadoop 1 vs hadoop 2 ,and apache community version vs cloudrea version,anyone can help?
答复： which hadoop version i can choose in production env?
What is MR1? -- 使用语盒发送 @2013-06-25 03:39 http://www.yuchs.com -- 原始邮件 -- user@hadoop.apache.org @2013年06月25日 03:27 I advice community version of Hadoop-1.1.2, which is a stable release, Hadoop2 hasn't stable release currently, even if all alpha release was extensive tested. but from me, I think HDFS2 is stable now.(no?), MR1 is also stable, but Yarn still need extensive tests(at least I think so), so our prod cluster using HDFS2 and MR1 now. On Tue, Jun 25, 2013 at 10:11 AM, ch huang justlo...@gmail.com wrote: hadoop 1 vs hadoop 2 ,and apache community version vs cloudrea version,anyone can help?
答复： which hadoop version i can choose in p roduction env?
I do not intent to use mapreduce ,I use hbase on hadoop -- 使用语盒发送 @2013-06-25 03:44 http://www.yuchs.com -- 原始邮件 -- user@hadoop.apache.org @2013年06月25日 03:36 Hi Yuhe, MR1 means Mapreduce version 1. (in Hadoop 1.x) On Tue, Jun 25, 2013 at 12:39 PM, yuhe justlo...@gmail.com wrote: What is MR1? -- 使用语盒发送 @2013-06-25 03:39 http://www.yuchs.com -- 原始邮件 -- user@hadoop.apache.org @2013年06月25日 03:27 I advice community version of Hadoop-1.1.2, which is a stable release, Hadoop2 hasn't stable release currently, even if all alpha release was extensive tested. but from me, I think HDFS2 is stable now.(no?), MR1 is also stable, but Yarn still need extensive tests(at least I think so), so our prod cluster using HDFS2 and MR1 now. On Tue, Jun 25, 2013 at 10:11 AM, ch huang justlo...@gmail.com wrote: hadoop 1 vs hadoop 2 ,and apache community version vs cloudrea version,anyone can help? -- -- Tatsuo Kawasaki tat...@cloudera.com
RE: After importing into Eclipse
It is a good start Lokesh. Can you go through this page http://wiki.apache.org/hadoop/HowToContribute for the steps and guidelines to contribute. Thanks Devaraj k From: Lokesh Basu [mailto:lokesh.b...@gmail.com] Sent: 25 June 2013 11:03 To: user@hadoop.apache.org Subject: After importing into Eclipse Hi all, I built the hdfs-trunk through maven and imported the project into eclipse through eclipse maven plugin. and then the project view shows the following list which is different from the content from my hadoop-trunk folder : hadoop-common-project hadoop-hdfs-project hadoop-main hadoop-mapreduce hadoop-mapreduce-client hadoop-pipes hadoop-project hadoop-project-dist hadoop-tools hadoop-yarn hadoop-yarn-applications hadoop-yarn-project hadoop-yarn-server hadoop-annotations hadoop-archives hadoop-assemblies hadoop-auth hadoop-auth-examples hadoop-client hadoop-common hadoop-datajoin hadoop-dist hadoop-distcp hadoop-extras hadoop-gridmix hadoop-hdfs hadoop-hdfs-bkjournal hadoop-hdfs-httpfs hadoop-mapreduce-client-app hadoop-mapreduce-client-common hadoop-mapreduce-client-core hadoop-mapreduce-client-hs hadoop-mapreduce-client-hs-plugins hadoop-mapreduce-client-jobclient hadoop-mapreduce-client-shuffle hadoop-mapreduce-examples hadoop-maven-plugins hadoop-minicluster hadoop-nfs hadoop-rumen hadoop-streaming hadoop-tools-dist hadoop-yarn-api hadoop-yarn-applications-distributedshell hadoop-yarn-applications-unmanaged-am-launcher hadoop-yarn-client hadoop-yarn-common hadoop-yarn-server-common hadoop-yarn-server-nodemanager hadoop-yarn-server-resourcemanager hadoop-yarn-server-tests hadoop-yarn-server-web-proxy hadoop-yarn-site I want to solve this [https://issues.apache.org/jira/browse/HDFS-336] bug which is for newcomer and want to get involved. I want to know about the ways to proceed from this point. Can anyone help me over this. -- Lokesh Basu
