Re: Zeppelin 0.8.0 with JDBC to Hive settings
BTW, please ask this kind of question in zeppelin user mail list instead of hadoop mail list. Jeff Zhang 于2018年10月25日周四 下午9:01写道： > >>> java.sql.SQLException: Could not open client transport with JDBC Uri: > jdbc:hive2://localhost:1: java.net.ConnectException: Connection refused > (Connection refused) > > It looks like due to hive connection issue, Can you first verify whether > you can connect to hive in beeline ? > > > Lee Ming-Ta 于2018年10月25日周四 下午6:01写道： > >> Dear all, I'm wondering whether anyone would help me with the settings >> for Zeppelin 0.8.0 using Hive. >> >> As indicated by Apache on Zeppelin website, the interpreter for Hive has >> been deprecated and merged with JDBC. However, I cannot use %jdbc(hive) to >> initiate it in Zeppelin. All I got is "InterpreterNotFoundException: >> Either no interpreter named hive or it is not binded to this note". >> >> I've tried following this website: >> https://zeppelin.apache.org/docs/0.8.0/interpreter/jdbc.html#apache-hive and >> change the interpreter settings for jdbc (except default.user and >> default.password). >> I also made two links: >> >> hadoop-common.jar -> >> /usr/local/hadoop/share/hadoop/common/hadoop-common-3.1.1.jar >> hive-jdbc.jar -> >> /usr/local/apache-hive-3.1.0-bin/jdbc/hive-jdbc-3.1.0-standalone.jar >> >> in /usr/local/zeppelin/interpreter/jdbc/. >> >> However, this time I got a different error: >> >> java.sql.SQLException: Could not open client transport with JDBC Uri: >> jdbc:hive2://localhost:1: java.net.ConnectException: Connection refused >> (Connection refused) >> >> Is there anything I shoud do? >> >> Thanks for help! >> >> Ming >> >
Re: Zeppelin 0.8.0 with JDBC to Hive settings
>>> java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:1: java.net.ConnectException: Connection refused (Connection refused) It looks like due to hive connection issue, Can you first verify whether you can connect to hive in beeline ? Lee Ming-Ta 于2018年10月25日周四 下午6:01写道： > Dear all, I'm wondering whether anyone would help me with the settings for > Zeppelin 0.8.0 using Hive. > > As indicated by Apache on Zeppelin website, the interpreter for Hive has > been deprecated and merged with JDBC. However, I cannot use %jdbc(hive) to > initiate it in Zeppelin. All I got is "InterpreterNotFoundException: > Either no interpreter named hive or it is not binded to this note". > > I've tried following this website: > https://zeppelin.apache.org/docs/0.8.0/interpreter/jdbc.html#apache-hive and > change the interpreter settings for jdbc (except default.user and > default.password). > I also made two links: > > hadoop-common.jar -> > /usr/local/hadoop/share/hadoop/common/hadoop-common-3.1.1.jar > hive-jdbc.jar -> > /usr/local/apache-hive-3.1.0-bin/jdbc/hive-jdbc-3.1.0-standalone.jar > > in /usr/local/zeppelin/interpreter/jdbc/. > > However, this time I got a different error: > > java.sql.SQLException: Could not open client transport with JDBC Uri: > jdbc:hive2://localhost:1: java.net.ConnectException: Connection refused > (Connection refused) > > Is there anything I shoud do? > > Thanks for help! > > Ming >
Hadoop 3.0 doesn't detect the correct conf folder
I tried the hadoop 3.0, and can start dfs properly, but when I start yarn, it fails with the following error ERROR: Cannot find configuration directory "/Users/jzhang/Java/lib/hadoop-3.0.0/conf" Actually, this is not the correct conf folder. It should be /Users/jzhang/Java/lib/hadoop-3.0.0/etc/hadoop. hdfs could detect that correctly, but seems yarn couldn't, might be something wrong in yarn starting script.
Re: Hadoop Issues - Can't find on StackOverflow
I would suggest user to run hadoop on linux, do not waste time on these OS related issues. zlgonzalez 于2017年7月23日周日 下午12:06写道： > Looks like you're running on Windows. Not sure if hadoop running on > Windows even for single nodes is still supported... > > Thanks, > Ron > > > > Sent via the Samsung Galaxy S7 active, an AT&T 4G LTE smartphone > > Original message > From: "संजीव (Sanjeev Tripurari)" > Date: 7/22/17 8:25 PM (GMT-08:00) > To: johnsonngu...@ups.com.invalid > Cc: user@hadoop.apache.org > Subject: Re: Hadoop Issues - Can't find on StackOverflow > > Hi, > > Can you share the environment where you did the setup. > - Which distribution > - OS > - Hadoop version > > Regards > -Sanjeev > > > On 19 July 2017 at 00:36, wrote: > >> Hi Hadoop People, >> >> >> >> I cannot get my YARN to run for my single node cluster. The error I >> receive when I run start-yarn.sh is: >> >> No Title >> >> starting yarn daemons >> >> mkdir: cannot create directory ‘/opt/hadoop/bin/yarn.cmd’: Not a directory >> >> chown: cannot access ‘/opt/hadoop/bin/yarn.cmd/logs’: Not a directory >> >> starting resourcemanager, logging to >> /opt/hadoop/bin/yarn.cmd/logs/yarn-vkq8pyw-resourcemanager-master.out >> >> /root/hadoop-2.7.2/sbin/yarn-daemon.sh: line 123: cd: >> /opt/hadoop/bin/yarn.cmd: Not a directory >> >> /root/hadoop-2.7.2/sbin/yarn-daemon.sh: line 124: >> /opt/hadoop/bin/yarn.cmd/logs/yarn-vkq8pyw-resourcemanager-master.out: Not >> a directory >> >> head: cannot open >> ‘/opt/hadoop/bin/yarn.cmd/logs/yarn-vkq8pyw-resourcemanager-master.out’ for >> reading: Not a directory >> >> /root/hadoop-2.7.2/sbin/yarn-daemon.sh: line 129: >> /opt/hadoop/bin/yarn.cmd/logs/yarn-vkq8pyw-resourcemanager-master.out: Not >> a directory >> >> /root/hadoop-2.7.2/sbin/yarn-daemon.sh: line 130: >> /opt/hadoop/bin/yarn.cmd/logs/yarn-vkq8pyw-resourcemanager-master.out: Not >> a directory >> >> >> >> >> >> If there is a fix to this, please do tell. Thank you for your help and >> reading. >> >> >> >> Best Regards, >> >> Johnson Nguyen >> > >
Re: Can hdfs client 2.6 read file of hadoop 2.7 ?
Thanks Ravi Ravi Prakash 于2017年6月27日周二 上午3:49写道： > Hi Jeff! > > Yes. hadoop-2.6 clients are able to read files on a hadoop-2.7 cluster. > The document I could find is > http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Compatibility.html > . > > "Both Client-Server and Server-Server compatibility is preserved within a > major release" > > HTH > Ravi. > > On Mon, Jun 26, 2017 at 5:21 AM, Jeff Zhang wrote: > >> >> It looks like it can. But is there any document about the compatibility >> between versions ? Thanks >> >> >> >
Can hdfs client 2.6 read file of hadoop 2.7 ?
It looks like it can. But is there any document about the compatibility between versions ? Thanks
Re: Configuration per job
No, this is RM configuration, which is applied to all jobs. 정현진 于2016年10月26日周三 上午7:23写道： > Hi. > > Is it possible to change property per Job in *-site.xml like* yarn*. > *scheduler*.minimum-allocation-*mb*? > > I want to set the value of* yarn*.*scheduler*.minimum-allocation-*mb* in > MapReduce application's main() function. > > Jung >
Re: how to use Yarn API to find task/attempt status
If it is for M/R, then maybe this is what you want https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/mapreduce/JobStatus.html On Thu, Mar 10, 2016 at 1:58 PM, Frank Luo wrote: > Let’s say there are 10 standard M/R jobs running. How to find how many > tasks are done/running/pending? > > > > *From:* Jeff Zhang [mailto:zjf...@gmail.com] > *Sent:* Wednesday, March 09, 2016 9:33 PM > *To:* Frank Luo > *Cc:* user@hadoop.apache.org > *Subject:* Re: how to use Yarn API to find task/attempt status > > > > I don't think it is related with yarn. Yarn don't know about task/task > attempt, it only knows containers. So it should be your application to > provide such function. > > > > On Thu, Mar 10, 2016 at 11:29 AM, Frank Luo wrote: > > Anyone had a similar issue and knows the answer? > > > > *From:* Frank Luo > *Sent:* Wednesday, March 09, 2016 1:59 PM > *To:* 'user@hadoop.apache.org' > *Subject:* how to use Yarn API to find task/attempt status > > > > I have a need to programmatically find out how many tasks are pending in > Yarn. Is there a way to do it through a Java API? > > > > I looked at YarnClient, but not able to find what I need. > > > > Thx in advance. > > > > Frank Luo > > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute > this email without the author’s prior permission. We take precautions to > minimize the risk of transmitting software viruses, but we advise you to > perform your own virus checks on any attachment to this message. We cannot > accept liability for any loss or damage caused by software viruses. The > information contained in this communication may be confidential and may be > subject to the attorney-client privilege. > > > > > > -- > > Best Regards > > Jeff Zhang > > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute > this email without the author’s prior permission. We take precautions to > minimize the risk of transmitting software viruses, but we advise you to > perform your own virus checks on any attachment to this message. We cannot > accept liability for any loss or damage caused by software viruses. The > information contained in this communication may be confidential and may be > subject to the attorney-client privilege. > -- Best Regards Jeff Zhang
Re: how to use Yarn API to find task/attempt status
I don't think it is related with yarn. Yarn don't know about task/task attempt, it only knows containers. So it should be your application to provide such function. On Thu, Mar 10, 2016 at 11:29 AM, Frank Luo wrote: > Anyone had a similar issue and knows the answer? > > > > *From:* Frank Luo > *Sent:* Wednesday, March 09, 2016 1:59 PM > *To:* 'user@hadoop.apache.org' > *Subject:* how to use Yarn API to find task/attempt status > > > > I have a need to programmatically find out how many tasks are pending in > Yarn. Is there a way to do it through a Java API? > > > > I looked at YarnClient, but not able to find what I need. > > > > Thx in advance. > > > > Frank Luo > > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute > this email without the author’s prior permission. We take precautions to > minimize the risk of transmitting software viruses, but we advise you to > perform your own virus checks on any attachment to this message. We cannot > accept liability for any loss or damage caused by software viruses. The > information contained in this communication may be confidential and may be > subject to the attorney-client privilege. > -- Best Regards Jeff Zhang
Re: Confusing Yarn RPC Configuration
Thanks, looks like it is resolved in 2.7 On Wed, Aug 19, 2015 at 3:03 PM, Rohith Sharma K S < rohithsharm...@huawei.com> wrote: > >>> I believe it is the same issue for node manage connection > > This would be probably related to below issues > > https://issues.apache.org/jira/i#browse/YARN-3944 > > https://issues.apache.org/jira/i#browse/YARN-3238 > > > > > > Thanks & Regards > > Rohith Sharma K S > > > > *From:* Jeff Zhang [mailto:zjf...@gmail.com] > *Sent:* 18 August 2015 09:11 > *To:* user@hadoop.apache.org > *Subject:* Confusing Yarn RPC Configuration > > > > > I use yarn.resourcemanager.connect.max-wait.ms to control how much time > to wait for setting up RM connection. But the weird thing I found that this > configuration is not the real max wait time. Actually Yarn will convert it > to retry count with configuration > yarn.resourcemanager.connect.retry-interval.ms. > > Let's say yarn.resourcemanager.connect.max-wait.ms=1 and > yarn.resourcemanager.connect.retry-interval.ms=2000, then yarn will > create RetryUpToMaximumCountWithFixedSleep with max count = 5 (1/2000) > > Because for each RM connection, there's retry policy inside of hadoop RPC. > Let's say ipc.client.connect.retry.interval=1000 > and ipc.client.connect.max.retries=10, so for each RM connection it will > try 10 times and totally cost 10 seconds (1000*10). So overall for the RM > connection it would cost 50 seconds (10 * 5), and this number is not > consistent with yarn.resourcemanager.connect.max-wait.ms which confuse > users. I am not sure the purpose of 2 rounds of retry policy (Yarn side and > RPC internal side), should it be only 1 round of retry policy and yarn > related configuration is just for override the RPC configuration ? > > > > BTW, I believe it is the same issue for node manage connection. > > > > -- > > Best Regards > > Jeff Zhang > -- Best Regards Jeff Zhang
Confusing Yarn RPC Configuration
I use yarn.resourcemanager.connect.max-wait.ms to control how much time to wait for setting up RM connection. But the weird thing I found that this configuration is not the real max wait time. Actually Yarn will convert it to retry count with configuration yarn.resourcemanager.connect.retry-interval.ms. Let's say yarn.resourcemanager.connect.max-wait.ms=1 and yarn.resourcemanager.connect.retry-interval.ms=2000, then yarn will create RetryUpToMaximumCountWithFixedSleep with max count = 5 (1/2000) Because for each RM connection, there's retry policy inside of hadoop RPC. Let's say ipc.client.connect.retry.interval=1000 and ipc.client.connect.max.retries=10, so for each RM connection it will try 10 times and totally cost 10 seconds (1000*10). So overall for the RM connection it would cost 50 seconds (10 * 5), and this number is not consistent with yarn.resourcemanager.connect.max-wait.ms which confuse users. I am not sure the purpose of 2 rounds of retry policy (Yarn side and RPC internal side), should it be only 1 round of retry policy and yarn related configuration is just for override the RPC configuration ? BTW, I believe it is the same issue for node manage connection. -- Best Regards Jeff Zhang
Re: Should AMRMClientAsync#CallbackHandler add method onAMCommand ?
Thanks Hitesh, I only found onShutdownRequest is called
when ApplicationAttemptNotFoundException happens.
AllocateResponse#AMCommand is never checked.
AMRMClientAsyncImpl#HeartbeatThread
try {
response = client.allocate(progress);
} catch (ApplicationAttemptNotFoundException e) {
handler.onShutdownRequest();
LOG.info("Shutdown requested. Stopping callback.");
return;
}
On Fri, Aug 14, 2015 at 4:37 AM, Hitesh Shah wrote:
> Please look at CallbackHandler::onShutdownRequest()
>
> thanks
> — Hitesh
>
> On Aug 13, 2015, at 6:55 AM, Jeff Zhang wrote:
>
> > I see that AllocateResponse has AMCommand which may request AM to resync
> or shutdown, but I don't see AMRMClientAsync#CallbackHandler has any method
> to handle that. Should AMRMClientAsync#CallbackHandler add method
> onAMCommand ?
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
>
>
--
Best Regards
Jeff Zhang
Should AMRMClientAsync#CallbackHandler add method onAMCommand ?
I see that AllocateResponse has AMCommand which may request AM to resync or shutdown, but I don't see AMRMClientAsync#CallbackHandler has any method to handle that. Should AMRMClientAsync#CallbackHandler add method onAMCommand ? -- Best Regards Jeff Zhang
Re: Application Master waits a long time after Mapper/Reducers finish
Might due to performance issue of FileOutputCommitter which is resolved in 2.7 https://issues.apache.org/jira/browse/MAPREDUCE-4815 Best Regard, Jeff Zhang From: Ashish Kumar Singh mailto:ashish23...@gmail.com>> Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" mailto:user@hadoop.apache.org>> Date: Monday, July 20, 2015 at 4:06 AM To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" mailto:user@hadoop.apache.org>> Subject: Re: Application Master waits a long time after Mapper/Reducers finish Hi Rohit , Thanks for replying . No , I do not see any connection retry attempts to HDFS in the logs . Also , Namenode and HDFS look healthy in our cluster . PFA latest AM logs for the job . Regards, Ashish On Mon, Jul 20, 2015 at 3:29 PM, Rohith Sharma K S mailto:rohithsharm...@huawei.com>> wrote: Hi >From thread dump, it seems waiting for HDFS operation. Can you attach AM >logs, and do you see any client retry for connecting to HDFS? "CommitterEvent Processor #4" prio=10 tid=0x0199a800 nid=0x18df in Object.wait() [0x7f4f12aa4000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:1864) at org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:575) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:345) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:362) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237) May be you can check from HDFS that is it Healthy? Thanks & Regards Rohith Sharma K S From: Ashish Kumar Singh [mailto:ashish23...@gmail.com<mailto:ashish23...@gmail.com>] Sent: 20 July 2015 14:16 To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: Application Master waits a long time after Mapper/Reducers finish Hello Users , I am facing a problem running Mapreduce jobs on Hadoop 2.6. I am observing that the Applocation Master waits for a long time after all the Mappers and Reducers are completed before the job is completed . This wait time sometimes exceeds 20-25 mins which is very strange as our mappers and reducers complete in less than 10 minutes for the job . Below are some observations: a) Job completion status stands at 95% when the wait begins b)JOB_COMMIT is initiated just before this wait time ( logs: 2015-07-14 01:54:46,636 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1436854849540_0123Job Transitioned from RUNNING to COMMITTING ) c) job success happens after 20-25 minutes ( logs: 2015-07-14 02:15:06,634 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1436854849540_0123Job Transitioned from COMMITTING to SUCCEEDED ) Appreciate any help on this . Thread dump while the Application master hangs is attached. Regards, Ashish
Is the application tracking URL changed expected ?
I run the distributed shell example, and the tracking url will change after app move to RUNNING state. As the following log shows that after the app move to RUNNING state, there's one "A" suffix in the trackingURL. But it looks like the suffix "A" is not used. Because even I change the suffix to any other words, it will redirect me to the right app report url. So I'm not sure the purpose of the tracking URL suffix ? Is this by design or a bug ? 15/05/06 21:14:53 INFO distributedshell.Client: Got application report from ASM for, appId=3, clientToAMToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1, appStartTime=1430918063457, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl= http://localhost:8088/proxy/application_1430916889869_0003/, appUser=jzhang 15/05/06 21:14:54 INFO distributedshell.Client: Got application report from ASM for, appId=3, clientToAMToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1, appStartTime=1430918063457, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl= http://localhost:8088/proxy/application_1430916889869_0003/, appUser=jzhang 15/05/06 21:14:55 INFO distributedshell.Client: Got application report from ASM for, appId=3, clientToAMToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default, appMasterRpcPort=-1, appStartTime=1430918063457, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, *appTrackingUrl=http://localhost:8088/proxy/application_1430916889869_0003/ <http://localhost:8088/proxy/application_1430916889869_0003/>,* appUser=jzhang 15/05/06 21:14:56 INFO distributedshell.Client: Got application report from ASM for, appId=3, clientToAMToken=null, appDiagnostics=, appMasterHost=jzhangMBPr.local/127.0.0.1, appQueue=default, appMasterRpcPort=-1, appStartTime=1430918063457, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, *appTrackingUrl=http://localhost:8088/proxy/application_1430916889869_0003/A <http://localhost:8088/proxy/application_1430916889869_0003/A>*, appUser=jzhang 15/05/06 21:14:57 INFO distributedshell.Client: Got application report from ASM for, appId=3, clientToAMToken=null, appDiagnostics=, appMasterHost=jzhangMBPr.local/127.0.0.1, appQueue=default, appMasterRpcPort=-1, appStartTime=1430918063457, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl= http://localhost:8088/proxy/application_1430916889869_0003/A, appUser=jzhang -- Best Regards Jeff Zhang
Invalid token issue in yarn
Sometimes my job will get the following error. What may be the reason for this ? And is there any property that I can use to prevent this ? Looks like someone got the same error. http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201502.mbox/%3c54e64f97.7070...@ulul.org%3E 2015-04-01 17:52:27,120 WARN [AMRM Heartbeater thread] ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): appattempt_1427881755072_0009_01 not found in AMRMTokenSecretManager. 2015-04-01 17:52:27,123 ERROR [AMRM Heartbeater thread] impl.AMRMClientAsyncImpl: Exception on heartbeat org.apache.hadoop.security.token.SecretManager$InvalidToken: appattempt_1427881755072_0009_01 not found in AMRMTokenSecretManager. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.allocate(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224) -- Best Regards Jeff Zhang
Re: Hadoop 2.6 issue
Try to export JAVA_HOME in hadoop-env.sh Best Regard, Jeff Zhang From: Anand Murali mailto:anand_vi...@yahoo.com>> Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" mailto:user@hadoop.apache.org>>, Anand Murali mailto:anand_vi...@yahoo.com>> Date: Wednesday, April 1, 2015 at 2:28 PM To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" mailto:user@hadoop.apache.org>> Subject: Hadoop 2.6 issue Dear All: I am unable to start Hadoop even after setting HADOOP_INSTALL,JAVA_HOME and JAVA_PATH. Please find below error message anand_vihar@Latitude-E5540:~/hadoop-2.6.0$ start-dfs.sh --config /home/anand_vihar/hadoop-2.6.0/conf Starting namenodes on [localhost] localhost: Error: JAVA_HOME is not set and could not be found. cat: /home/anand_vihar/hadoop-2.6.0/conf/slaves: No such file or directory Starting secondary namenodes [0.0.0.0] 0.0.0.0: Error: JAVA_HOME is not set and could not be found. anand_vihar@Latitude-E5540:~/hadoop-2.6.0$ echo $JAVA_HOME /usr/lib/jvm/java-1.7.0-openjdk-amd64 anand_vihar@Latitude-E5540:~/hadoop-2.6.0$ echo $HADOOP_INSTALL /home/anand_vihar/hadoop-2.6.0 anand_vihar@Latitude-E5540:~/hadoop-2.6.0$ echo $PATH :/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/anand_vihar/hadoop-2.6.0/bin:/home/anand_vihar/hadoop-2.6.0/sbin:/usr/lib/jvm/java-1.7.0-openjdk-amd64:/usr/lib/jvm/java-1.7.0-openjdk-amd64 anand_vihar@Latitude-E5540:~/hadoop-2.6.0$ I HAVE MADE NO CHANGES IN HADOOP_ENV.sh and run it succesfully. Core-site.xml fs.default.name hdfs://localhost/ HDFS-site.xml dfs.replication 1 Mapred-site.xml mapred.job.tracker localhost:8021 Shall be thankful, if somebody can advise. Regards, Anand Murali 11/7, 'Anand Vihar', Kandasamy St, Mylapore Chennai - 600 004, India Ph: (044)- 28474593/ 43526162 (voicemail)
Re: Yarn setup problems
nerExecutor: at > java.util.concurrent.FutureTask.run(FutureTask.java:262) > 2015-03-19 21:40:00,958 INFO > org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > 2015-03-19 21:40:00,958 INFO > org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > 2015-03-19 21:40:00,958 INFO > org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at > java.lang.Thread.run(Thread.java:745) > > > I was trying to find out what went wrong, but couldn't find anything. I > have similiar cluster with the same configuration which runs YARN too > without any problems... > > Best regards, > > Morbious > -- Best Regards Jeff Zhang
Re: trying to understand HashPartitioner
You can take it similar as the HashMap of java. Use the hashCode of one object to distribute it into different bucket. Best Regard, Jeff Zhang From: xeonmailinglist-gmail mailto:xeonmailingl...@gmail.com>> Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" mailto:user@hadoop.apache.org>> Date: Wednesday, March 18, 2015 at 7:08 PM To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" mailto:user@hadoop.apache.org>> Subject: Re: trying to understand HashPartitioner What tells with partition will run on which reduce task? On 18-03-2015 09:30, xeonmailinglist-gmail wrote: Hi, I am trying to understand how HashPartitioner.java works. Thus, I ran a mapreduce job with 5 reducers and 5 input files. I thought that the output of getPartition(K2 key, V2 value, int numReduceTasks) was the number of reduce task that K2 and V2 will execute. Is this correct? -- -- -- --
Re: Localizer failed but it works after I restart yarn
BTW, I am using hadoop 2.6 and it's a single node cluster on mac. On Fri, Feb 6, 2015 at 11:35 AM, Jeff Zhang wrote: > I intermittently meet this issue of Localizer failed, but after I restart > the yarn, then the problem is gone. Is this a known issue ? > > Here's log in node manager > > > 2015-02-06 11:31:24,898 INFO > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > Localizer CWD set to > /tmp/hadoop-jzhang/nm-local-dir/usercache/jzhang/appcache/application_1423126607079_0007 > = > file:/tmp/hadoop-jzhang/nm-local-dir/usercache/jzhang/appcache/application_1423126607079_0007 > 2015-02-06 11:31:24,906 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > Localizer failed > java.lang.NullPointerException > at java.util.zip.ZipFile.getZipEntry(ZipFile.java:531) > at java.util.zip.ZipFile.access$900(ZipFile.java:56) > at java.util.zip.ZipFile$1.nextElement(ZipFile.java:513) > at java.util.zip.ZipFile$1.nextElement(ZipFile.java:483) > at java.util.jar.JarFile$1.nextElement(JarFile.java:244) > at java.util.jar.JarFile$1.nextElement(JarFile.java:239) > at sun.misc.URLClassPath$JarLoader.validIndex(URLClassPath.java:803) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:937) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:848) > at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818) > at sun.misc.URLClassPath.findResource(URLClassPath.java:176) > at java.net.URLClassLoader$2.run(URLClassLoader.java:551) > at java.net.URLClassLoader$2.run(URLClassLoader.java:549) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findResource(URLClassLoader.java:548) > at java.lang.ClassLoader.getResource(ClassLoader.java:1147) > at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:227) > at org.apache.xerces.parsers.SecuritySupport$6.run(Unknown Source) > at java.security.AccessController.doPrivileged(Native Method) > at > org.apache.xerces.parsers.SecuritySupport.getResourceAsStream(Unknown > Source) > at > org.apache.xerces.parsers.ObjectFactory.findJarServiceProvider(Unknown > Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) > at > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > Source) > at > org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2402) > at > org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2364) > at > org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2281) > at org.apache.hadoop.conf.Configuration.set(Configuration.java:1019) > at org.apache.hadoop.conf.Configuration.set(Configuration.java:991) > at > org.apache.hadoop.conf.Configuration.setStrings(Configuration.java:1791) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:407) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:129) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:129) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1088) > > -- > Best Regards > > Jeff Zhang > -- Best Regards Jeff Zhang
Localizer failed but it works after I restart yarn
I intermittently meet this issue of Localizer failed, but after I restart the yarn, then the problem is gone. Is this a known issue ? Here's log in node manager 2015-02-06 11:31:24,898 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-jzhang/nm-local-dir/usercache/jzhang/appcache/application_1423126607079_0007 = file:/tmp/hadoop-jzhang/nm-local-dir/usercache/jzhang/appcache/application_1423126607079_0007 2015-02-06 11:31:24,906 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed java.lang.NullPointerException at java.util.zip.ZipFile.getZipEntry(ZipFile.java:531) at java.util.zip.ZipFile.access$900(ZipFile.java:56) at java.util.zip.ZipFile$1.nextElement(ZipFile.java:513) at java.util.zip.ZipFile$1.nextElement(ZipFile.java:483) at java.util.jar.JarFile$1.nextElement(JarFile.java:244) at java.util.jar.JarFile$1.nextElement(JarFile.java:239) at sun.misc.URLClassPath$JarLoader.validIndex(URLClassPath.java:803) at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:937) at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:848) at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:818) at sun.misc.URLClassPath.findResource(URLClassPath.java:176) at java.net.URLClassLoader$2.run(URLClassLoader.java:551) at java.net.URLClassLoader$2.run(URLClassLoader.java:549) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findResource(URLClassLoader.java:548) at java.lang.ClassLoader.getResource(ClassLoader.java:1147) at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:227) at org.apache.xerces.parsers.SecuritySupport$6.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.xerces.parsers.SecuritySupport.getResourceAsStream(Unknown Source) at org.apache.xerces.parsers.ObjectFactory.findJarServiceProvider(Unknown Source) at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) at org.apache.xerces.parsers.DOMParser.(Unknown Source) at org.apache.xerces.parsers.DOMParser.(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2402) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2364) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2281) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1019) at org.apache.hadoop.conf.Configuration.set(Configuration.java:991) at org.apache.hadoop.conf.Configuration.setStrings(Configuration.java:1791) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:407) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:129) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:129) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1088) -- Best Regards Jeff Zhang
Re: reg pig issue
Hi mallik, You miss ":" between fieldname and field type, it should be date_d : chararray On Mon, Oct 20, 2014 at 12:50 PM, mallik arjun wrote: > HI hadoopers, > > > i have the following issue with pig > > > > > > raw_log_entries = LOAD '/user/u377058/rc_app_error' USING TextLoader AS > (line:chararray); > > logs_base = FOREACH raw_log_entries GENERATE > FLATTEN( > REGEX_EXTRACT_ALL(line, > '^(\\d+-\\d+-\\d+)\\s+(\\d+:\\d+:\\d+,\\d+)\\s+(\\w+)\\s+(\\w+-\\w+-\\d+-\\w+-\\d)\\s+([^\\s]+)\\s+:\\s+(.*)$' > ) > ) AS ( > date_d chararray, > timestamp_ts chararray, > event chararray, > error_msg chararray, > package chararray, > separator chararray, > description chararray > ); > > > > when exeucting the above porgram following issue is coming. > > Unexpected character 'D' > 2014-10-19 23:09:12,549 [main] ERROR org.apache.pig.tools.grunt.Grunt - > ERROR 1200: Unexpected > character 'D' > Details at logfile: /home/u377058/pig_1413774551268.log > > > please help us > > thanks in advance. > > > > -- Best Regards Jeff Zhang
Re: Spark vs Tez
Tez has a feature called pre-warm which will launch JVM before you use it and you can reuse the container afterwards. So it is also suitable for interactive queries and is more stable and scalable than spark IMO. On Sat, Oct 18, 2014 at 4:22 PM, Niels Basjes wrote: > It is my understanding that one of the big differences between Tez and > Spark is is that a Tez based query still has the startup overhead of > starting JVMs on the Yarn cluster. Spark based queries are immediately > executed on "already running JVMs". > > So for interactive dashboards Spark seems more suitable. > > Did I understand correctly? > > Niels Basjes > On Oct 17, 2014 8:30 PM, "Gavin Yue" wrote: > >> Spark and tez both make MR faster, this has no doubt. >> >> They also provide new features like DAG, which is quite important for >> interactive query processing. From this perspective, you could view them >> as a wrapper around MR and try to handle the intermediary buffer(files) >> more efficiently. It is a big pain in MR. >> >> Also they both try to use Memory as the buffer instead of only >> filesystems. Spark has a concept RDD, which is quite interesting and also >> limited. >> >> >> >> On Fri, Oct 17, 2014 at 11:23 AM, Adaryl "Bob" Wakefield, MBA < >> adaryl.wakefi...@hotmail.com> wrote: >> >>> It was my understanding that Spark is faster batch processing. Tez is >>> the new execution engine that replaces MapReduce and is also supposed to >>> speed up batch processing. Is that not correct? >>> B. >>> >>> >>> >>> *From:* Shahab Yunus >>> *Sent:* Friday, October 17, 2014 1:12 PM >>> *To:* user@hadoop.apache.org >>> *Subject:* Re: Spark vs Tez >>> >>> What aspects of Tez and Spark are you comparing? They have different >>> purposes and thus not directly comparable, as far as I understand. >>> >>> Regards, >>> Shahab >>> >>> On Fri, Oct 17, 2014 at 2:06 PM, Adaryl "Bob" Wakefield, MBA < >>> adaryl.wakefi...@hotmail.com> wrote: >>> >>>> Does anybody have any performance figures on how Spark stacks up >>>> against Tez? If you don’t have figures, does anybody have an opinion? Spark >>>> seems so popular but I’m not really seeing why. >>>> B. >>>> >>> >>> >> >> -- Best Regards Jeff Zhang
Re: SIMPLE authentication is not enabled. Available:[TOKEN]
Here's my sample for your reference ( If you are running unmanaged AM in
client side):
1. set token in UserGroupInformation
2. do the registration in the way of UserGroupInformation as following
try {
ugi.addToken(yarnClient.getAMRMToken(appId));
ugi.doAs(new PrivilegedExceptionAction() {
@Override
public Void run() throws Exception {
AMRMClient amRMClient = AMRMClient
.createAMRMClient();
amRMClientAsync =
AMRMClientAsync.createAMRMClientAsync(amRMClient,
200, new InnerCallbackHandler());
amRMClientAsync.init(conf);
amRMClientAsync.start();
amRMClientAsync.registerApplicationMaster("localhost", 0, "url");
return null;
}
});
} catch (Throwable ex) {
ex.printStackTrace();
}
On Sun, Mar 16, 2014 at 8:19 PM, Oleg Zhurakousky <
oleg.zhurakou...@gmail.com> wrote:
> Thanks Jeff
>
> Yes I am using 2.3 and the issue is still there.
>
> Oleg
>
>
> On Sun, Mar 16, 2014 at 3:10 AM, Jeff Zhang wrote:
>
>> Hi Oleg,
>>
>> I meet the same issue when I start an unmanaged AM in client side in
>> thread way. The issue is in the code of hadoop-common-yarn. You could try
>> to use the code of hadoop-common-yarn of 2.3 instead of 2.2 This resolve
>> my problem at least.
>>
>>
>>
>>
>> On Sun, Mar 16, 2014 at 5:56 AM, Oleg Zhurakousky <
>> oleg.zhurakou...@gmail.com> wrote:
>>
>>> The bug you referring to is this i think
>>>
>>> https://issues.apache.org/jira/browse/YARN-945
>>>
>>> . . . and is not really the issue in my case, since my IPv6 is disabled.
>>>
>>> What I really want to know is why TOKEN is hard coded in ipc Server. I
>>> can't wait to hear the argument ;)
>>>
>>>
>>> Anyway, thanks for reply.
>>>
>>> Oleg
>>>
>>>
>>>
>>>
>>> On Sat, Mar 15, 2014 at 5:19 PM, Edward Capriolo
>>> wrote:
>>>
>>>> There was a bug around this message that was fixed. I found another bug
>>>> I forgot to report. If your system is using ipv6 yarn get get confused over
>>>> the source/destination ips and throw this message at you. You can go in
>>>> your configuration files and specific a specfic address to bind. You can
>>>> also go into your hostfile and ensure localhost does not refer to an ipv6
>>>> address. Java also has a -D switch like preferIPV4 or something like that.
>>>>
>>>>
>>>> On Sat, Mar 15, 2014 at 4:18 PM, Oleg Zhurakousky <
>>>> oleg.zhurakou...@gmail.com> wrote:
>>>>
>>>>> So here is my dilemma.
>>>>>
>>>>> I am trying to register ApplicationMaster to a remote YARN cluster and
>>>>> I get
>>>>>
>>>>> Caused by:
>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
>>>>> SIMPLE authentication is not enabled. Available:[TOKEN]
>>>>>
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1406)
>>>>>
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1359)
>>>>>
>>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
>>>>> ProtobufRpcEngine.java:206)
>>>>>
>>>>> at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(
>>>>> ApplicationMasterProtocolPBClientImpl.java:106)
>>>>> My security is explicitly set to SIMPLE and I can see it in my logs:
>>>>>
>>>>> org.apache.hadoop.ipc.Server: Server accepts auth methods:[TOKEN,
>>>>> SIMPLE]
>>>>>
>>>>> What is unclear is why [TOKEN] get's there and when I say unclear, its
>>>>> actually very clear from the ipc Server code. The real question is why
>>>>> TOKEN is hard coded there in the first place for the cases when Secret
>>>>> Manager is present. Which also raises another question; How to disable
>>>>> Secret Manager.
>>>>>
>>>>> Cheers
>>>>> Oleg
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>
--
Best Regards
Jeff Zhang
Re: SIMPLE authentication is not enabled. Available:[TOKEN]
Hi Oleg, I meet the same issue when I start an unmanaged AM in client side in thread way. The issue is in the code of hadoop-common-yarn. You could try to use the code of hadoop-common-yarn of 2.3 instead of 2.2 This resolve my problem at least. On Sun, Mar 16, 2014 at 5:56 AM, Oleg Zhurakousky < oleg.zhurakou...@gmail.com> wrote: > The bug you referring to is this i think > > https://issues.apache.org/jira/browse/YARN-945 > > . . . and is not really the issue in my case, since my IPv6 is disabled. > > What I really want to know is why TOKEN is hard coded in ipc Server. I > can't wait to hear the argument ;) > > > Anyway, thanks for reply. > > Oleg > > > > > On Sat, Mar 15, 2014 at 5:19 PM, Edward Capriolo wrote: > >> There was a bug around this message that was fixed. I found another bug I >> forgot to report. If your system is using ipv6 yarn get get confused over >> the source/destination ips and throw this message at you. You can go in >> your configuration files and specific a specfic address to bind. You can >> also go into your hostfile and ensure localhost does not refer to an ipv6 >> address. Java also has a -D switch like preferIPV4 or something like that. >> >> >> On Sat, Mar 15, 2014 at 4:18 PM, Oleg Zhurakousky < >> oleg.zhurakou...@gmail.com> wrote: >> >>> So here is my dilemma. >>> >>> I am trying to register ApplicationMaster to a remote YARN cluster and I >>> get >>> >>> Caused by: >>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): >>> SIMPLE authentication is not enabled. Available:[TOKEN] >>> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1406) >>> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1359) >>> >>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke( >>> ProtobufRpcEngine.java:206) >>> >>> at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source) >>> >>> at >>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster( >>> ApplicationMasterProtocolPBClientImpl.java:106) >>> My security is explicitly set to SIMPLE and I can see it in my logs: >>> >>> org.apache.hadoop.ipc.Server: Server accepts auth methods:[TOKEN, SIMPLE] >>> >>> What is unclear is why [TOKEN] get's there and when I say unclear, its >>> actually very clear from the ipc Server code. The real question is why >>> TOKEN is hard coded there in the first place for the cases when Secret >>> Manager is present. Which also raises another question; How to disable >>> Secret Manager. >>> >>> Cheers >>> Oleg >>> >>> >> > -- Best Regards Jeff Zhang
Re: Using resource manager as standalone component from hadoop yarn
You can start the resource manager without starting any node manager. And the source code of resource manager and node manager are in different sub pom project. On Wed, Mar 12, 2014 at 7:06 AM, anu238 . wrote: > Hi All, > I am sorry for the blast, I wanted to use only the resource manager from > the hadoop yarn code base for building my platform for my use case, I was > wondering if anyone here have used only the resource manager from hadoop > yarn code base and how did they separated the resource manager component > from the other code. > > Thanks & Regards, > Anurag >
Re: Impact of Tez/Spark to MapReduce
I believe in the future the spark functional style api will dominate the big data world. Very few people will use the native mapreduce API. Even now usually users use third-party mapreduce library such as cascading, scalding, scoobi or script language hive, pig rather than the native mapreduce api. And this functional style of api compatible both with hadoop's mapreduce and spark's RDD. The underlying execution engine will be transparent to users. So I guess or I hope in the future, the api will be unified while the underlying execution engine will been choose intelligently according the resources you have and the metadata of the data you operate on. On Thu, Mar 6, 2014 at 9:02 AM, Edward Capriolo wrote: > The thing about yarn is you chose what is right for the the workload. > > For example: Spark may not the right choice if for example join tables do > not fit in memory. > > > On Wednesday, March 5, 2014, Anthony Mattas wrote: > > With Tez and Spark becoming mainstream what does Map Reduce look like > longer term? Will it become a component that sits on top of Tez, or will > they continue to live side by side utilizing YARN? > > I'm struggling a little bit to understand what the roadmap looks like > for the technologies that sit on top of YARN. > > > > Anthony Mattas > > anth...@mattas.net > > -- > Sorry this was sent from mobile. Will do less grammar and spell check than > usual. >
Re: Question on the UnManaged-AM-Launcher
I guess I got the answer , the reason is that we need to pass the environment to AM, and there's no way to pass that to thread, but it is possible for process. Could anyone confirm that ? On Wed, Feb 26, 2014 at 3:52 PM, Jeff Zhang wrote: > Hi all, > > I look the source code of unmanaged-am-launcher, and found that it will > launch a process for AM in client side. But it is still not easy to debug a > the AM since it is a process. Does it make more sense to run it in a > thread then it is more easy to debug in IDE ? Thanks > > > Jeff Zhang > >
Re: Logic of isSplittable() of class FileInputFormat
Hi Sugandha, Take gz file as an example, It is not splittable because of the compression algorithm it is used. It can not guarantee that one record is located in one block, if one record is in 2 blocks, your program will crash since you can not get the whole record. On Wed, Feb 26, 2014 at 1:24 PM, Sugandha Naolekar wrote: > Hello, > > If a single file is split of size 129 MB is split in two halves/blocks of > HDFS as the max block size id 128 MB. And each of the blocks is read > depending on the InputFormat it supports. Thus, what is the significance of > isSplittable() method then? > > If it is set to false, entire block will be considered as single input > split? How will TextInputFormat react to it? > > > -- > Thanks & Regards, > Sugandha Naolekar > > > >
Question on the UnManaged-AM-Launcher
Hi all, I look the source code of unmanaged-am-launcher, and found that it will launch a process for AM in client side. But it is still not easy to debug a the AM since it is a process. Does it make more sense to run it in a thread then it is more easy to debug in IDE ? Thanks Jeff Zhang
Unmanaged application issue on YARN
Hi all, I build an ummanaged application and submit it to yarn ( hadoop 2.2). But encounter the following exception: Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled. Available:[TOKEN] at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke( ProtobufRpcEngine.java:206) at $Proxy9.registerApplicationMaster(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster( ApplicationMasterProtocolPBClientImpl.java:106) I find there's one jira ticket very similar to this issue, but looks like it has been resolved in 2.2 https://issues.apache.org/jira/browse/YARN-945 I find that ResourceTrackerService will use simple authentication but ApplicationMasterService will use token. The reason I think is at the following code snippet in ApplicationMasterSerivce ( line 127 ) serverConf.set( CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, SaslRpcServer.AuthMethod.TOKEN.toString()); I'm not sure why here use the token, could any help explain that and guide me how to resolve my issue? Thanks Jeff Zhang
Re: question about hadoop dfs
you can use the fsck command to find the block locations, here's one example hadoop fsck /user/hadoop/graph_data.txt -blocks -locations -files On Sun, Jan 26, 2014 at 2:48 PM, EdwardKing wrote: > hdfs-site.xm is follows: > > > dfs.name.dir > file:/home/software/name > > > > dfs.namenode.secondary.http-address > master:9001 > > > dfs.data.dir > file:/home/software/data > > > dfs.http.address > master:9002 > > > dfs.replication > 2 > > > dfs.datanode.du.reserved > 1073741824 > > > dfs.block.size > 134217728 > > > dfs.permissions > false > > > > [root@master ~]# cd /home > [root@master home]# cd software/ > [root@master software]# ls > data hadoop-2.2.0 jdk1.7.0_02name test.txt > file: hadoop-2.2.0.tar.gz jdk-7u2-linux-i586.tar.gz temp tmp > > [root@master name]# pwd > /home/software/name > [root@master name]# ls > current in_use.lock > [root@master name]# > > [root@master software]# pwd > /home/software > [root@master software]# cd data > [root@master data]# ls > current in_use.lock > > >> the meadata(file name, file path and block location) is in master, the > file data itself is in datanode. > > Where I can find abc.txt meadata,such as file name, file path and block > location? The abc.txt file data itself is in master 172.11.12.6 or node1 > 172.11.12.7,which directory it locate? > > Thanks. > > > > - Original Message - > From: Jeff Zhang > To: user@hadoop.apache.org > Sent: Sunday, January 26, 2014 2:30 PM > Subject: Re: question about hadoop dfs > > > 1. Is supergroup a directory? Where does it locate? > supergroup is user group rather than directory just like the user > group of linux > > > 2. I search abc.txt on master 172.11.12.6 and node1 172.11.12.7 by > following command: > the meadata(file name, file path and block location) is in master, the > file data itself is in datanode. > > > > > > On Sun, Jan 26, 2014 at 2:22 PM, EdwardKing wrote: > > I use Hadoop2.2.0 to create a master node and a sub node,like follows: > > Live Datanodes : 2 > Node Transferring Address Last Contact Admin State Configured Capacity > (GB) Used(GB) Non DFS Used (GB) Remaining(GB) Used(%) > master 172.11.12.6:50010 1In Service > 16.15 0.00 2.76 > 13.39 0.00 > node1 172.11.12.7:50010 0 In Service > 16.15 0.00 2.75 > 13.40 0.00 > > Then I create a abc.txt file on master 172.11.12.6 > [hadoop@master ~]$ pwd > /home/hadoop > [hadoop@master ~]$ echo "This is a test." >> abc.txt > [hadoop@master ~]$ hadoop dfs -copyFromLocal test.txt > [hadoop@master ~]$ hadoop dfs -ls > DEPRECATED: Use of this script to execute hdfs command is deprecated. > Instead use the hdfs command for it. > > 14/01/25 22:07:00 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 2 hadoop supergroup 16 2014-01-25 21:36 abc.txt > > [hadoop@master ~]$ rm abc.txt > [hadoop@master ~]$ hadoop dfs -cat abc.txt > This is a test. > > My question is: > 1. Is supergroup a directory? Where does it locate? > 2. I search abc.txt on master 172.11.12.6 and node1 172.11.12.7 by > following command: > [hadoop@master ~]$ find / -name abc.txt > But I don't find abc.txt file. Where is the file abc.txt? After I erase it > by rm command, I still cat this file? Where is it? My OS is CentOS-5.8. > > Thanks. > > --- > Confidentiality Notice: The information contained in this e-mail and any > accompanying attachment(s) > is intended only for the use of the intended recipient and may be > confidential and/or privileged of > Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader > of this communication is > not the intended recipient, unauthorized use, forwarding, printing, > storing, disclosure or copying > is strictly prohibited, and may be unlawful.If you have received this > communication in error,please > immediately notify the sender by return e-mail, and delete the original > message and all copies from > your system. Thank you. > > --- > > ---
Re: question about hadoop dfs
1. Is supergroup a directory? Where does it locate? supergroup is user group rather than directory just like the user group of linux 2. I search abc.txt on master 172.11.12.6 and node1 172.11.12.7 by following command: the meadata(file name, file path and block location) is in master, the file data itself is in datanode. On Sun, Jan 26, 2014 at 2:22 PM, EdwardKing wrote: > I use Hadoop2.2.0 to create a master node and a sub node,like follows: > > Live Datanodes : 2 > Node Transferring Address Last Contact Admin State Configured Capacity > (GB) Used(GB) Non DFS Used (GB) Remaining(GB) Used(%) > master 172.11.12.6:50010 1In Service > 16.15 0.00 2.76 > 13.39 0.00 > node1 172.11.12.7:50010 0 In Service > 16.15 0.00 2.75 > 13.40 0.00 > > Then I create a abc.txt file on master 172.11.12.6 > [hadoop@master ~]$ pwd > /home/hadoop > [hadoop@master ~]$ echo "This is a test." >> abc.txt > [hadoop@master ~]$ hadoop dfs -copyFromLocal test.txt > [hadoop@master ~]$ hadoop dfs -ls > DEPRECATED: Use of this script to execute hdfs command is deprecated. > Instead use the hdfs command for it. > > 14/01/25 22:07:00 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Found 1 items > -rw-r--r-- 2 hadoop supergroup 16 2014-01-25 21:36 abc.txt > > [hadoop@master ~]$ rm abc.txt > [hadoop@master ~]$ hadoop dfs -cat abc.txt > This is a test. > > My question is: > 1. Is supergroup a directory? Where does it locate? > 2. I search abc.txt on master 172.11.12.6 and node1 172.11.12.7 by > following command: > [hadoop@master ~]$ find / -name abc.txt > But I don't find abc.txt file. Where is the file abc.txt? After I erase it > by rm command, I still cat this file? Where is it? My OS is CentOS-5.8. > > Thanks. > > --- > Confidentiality Notice: The information contained in this e-mail and any > accompanying attachment(s) > is intended only for the use of the intended recipient and may be > confidential and/or privileged of > Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader > of this communication is > not the intended recipient, unauthorized use, forwarding, printing, > storing, disclosure or copying > is strictly prohibited, and may be unlawful.If you have received this > communication in error,please > immediately notify the sender by return e-mail, and delete the original > message and all copies from > your system. Thank you. > > --- >
Re: [YARN] TestDistributedShell failed in eclipse but successful in maven
Thanks, Vinod. It works. Could you point me which part needs the JAVA_HOME env ? On Thu, Jan 23, 2014 at 2:17 AM, Vinod Kumar Vavilapalli < vino...@hortonworks.com> wrote: > Yes. It depends on JAVA_HOME env variable. When you run it, it will > succeed if you set that variable up correctly in eclipse test Run > configuration. > > +Vinod > > On Jan 21, 2014, at 5:39 PM, Jeff Zhang wrote: > > Hi all, > > TestDistributedShell is a unit test for DistributedShell. I could run it > successfully in maven, but when I run it in eclipse, it failed. Do I need > any extra setting to make it run in eclipse ? > > Here's the error message: > > 2014-01-22 09:38:20,733 INFO [AsyncDispatcher event handler] > rmapp.RMAppImpl (RMAppImpl.java:transition(788)) - Application > application_1390354688375_0001 failed 2 times due to AM Container for > appattempt_1390354688375_0001_02 exited with exitCode: 127 due to: > Exception from container-launch: > > org.apache.hadoop.util.Shell$ExitCodeException: > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) > > at org.apache.hadoop.util.Shell.run(Shell.java:379) > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute( > Shell.java:589) > > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( > DefaultContainerExecutor.java:195) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( > ContainerLaunch.java:283) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.EnhancedContainerLaunch.call( > EnhancedContainerLaunch.java:139) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.EnhancedContainerLaunch.call( > EnhancedContainerLaunch.java:1) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask( > ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:680) > > > > .Failing this attempt.. Failing the application. > > 2014-01-22 09:38:20,735 INFO [AsyncDispatcher event handler] > recovery.RMStateStore (RMStateStore.java:handleStoreEvent(487)) - > Removing info for app: application_1390354688375_0001 > > 2014-01-22 09:38:20,735 INFO [AsyncDispatcher event handler] > rmapp.RMAppImpl (RMAppImpl.java:handle(578)) - > application_1390354688375_0001 State change from ACCEPTED to FAILED > > 2014-01-22 09:38:20,736 INFO [ResourceManager Event Processor] > scheduler.AppSchedulingInfo (AppSchedulingInfo.java:clearRequests(108)) - > Application application_1390354688375_0001 requests cleared > > 2014-01-22 09:38:20,736 WARN [AsyncDispatcher event handler] > resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(255)) - > USER=zhangj82 OPERATION=Application Finished - Failed TARGET=RMAppManager > RESULT=FAILURE DESCRIPTION=App failed with state: FAILED > PERMISSIONS=Application > application_1390354688375_0001 failed 2 times due to AM Container for > appattempt_1390354688375_0001_02 exited with exitCode: 127 due to: > Exception from container-launch: > > org.apache.hadoop.util.Shell$ExitCodeException: > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) > > at org.apache.hadoop.util.Shell.run(Shell.java:379) > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute( > Shell.java:589) > > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( > DefaultContainerExecutor.java:195) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( > ContainerLaunch.java:283) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.EnhancedContainerLaunch.call( > EnhancedContainerLaunch.java:139) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.EnhancedContainerLaunch.call( > EnhancedContainerLaunch.java:1) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask( > ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:680) > > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
[YARN] TestDistributedShell failed in eclipse but successful in maven
Hi all, TestDistributedShell is a unit test for DistributedShell. I could run it successfully in maven, but when I run it in eclipse, it failed. Do I need any extra setting to make it run in eclipse ? Here's the error message: 2014-01-22 09:38:20,733 INFO [AsyncDispatcher event handler] rmapp.RMAppImpl (RMAppImpl.java:transition(788)) - Application application_1390354688375_0001 failed 2 times due to AM Container for appattempt_1390354688375_0001_02 exited with exitCode: 127 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.EnhancedContainerLaunch.call( EnhancedContainerLaunch.java:139) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.EnhancedContainerLaunch.call( EnhancedContainerLaunch.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask( ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) .Failing this attempt.. Failing the application. 2014-01-22 09:38:20,735 INFO [AsyncDispatcher event handler] recovery.RMStateStore (RMStateStore.java:handleStoreEvent(487)) - Removing info for app: application_1390354688375_0001 2014-01-22 09:38:20,735 INFO [AsyncDispatcher event handler] rmapp.RMAppImpl (RMAppImpl.java:handle(578)) - application_1390354688375_0001 State change from ACCEPTED to FAILED 2014-01-22 09:38:20,736 INFO [ResourceManager Event Processor] scheduler.AppSchedulingInfo (AppSchedulingInfo.java:clearRequests(108)) - Application application_1390354688375_0001 requests cleared 2014-01-22 09:38:20,736 WARN [AsyncDispatcher event handler] resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(255)) - USER=zhangj82 OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1390354688375_0001 failed 2 times due to AM Container for appattempt_1390354688375_0001_02 exited with exitCode: 127 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer( DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call( ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.EnhancedContainerLaunch.call( EnhancedContainerLaunch.java:139) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.EnhancedContainerLaunch.call( EnhancedContainerLaunch.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask( ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680)
Re: HIVE+MAPREDUCE
you just need to run mapreduce job to generate the data you want and then upload the data into hive table ( create table first if it is not exists ) these 2 steps are totally separated. On Tue, Jan 21, 2014 at 4:21 PM, Ranjini Rathinam wrote: > Hi, > > Need to load the data into hive table using mapreduce, using java. > > Please suggest the code related to hive +mapreduce. > > > > Thanks in advance > > Ranjini R > > >
Questions on AuxiliaryService
HI all, I found that the AuxiliaryService's implementation is a little confusing to me. The initializeApplication will be invoked on each container of one application including the am. While stopApplication will only been invoked on am when the job is done. It looks like there's a little mismatch between these 2 methods, could anyone explain to me why design it like this ? Thanks
Re: issue about total input byte of MR job
It depend on your input data. E.g. your input consists of 10 files, each is 65M, then each file will take 2 mappers, overall it would cost 20 mappers, but the input size is actually 650M rather than 20*64=1280M On Tue, Dec 3, 2013 at 4:28 PM, ch huang wrote: > i run the MR job,at the MR output i see > > 13/12/03 14:02:28 INFO mapreduce.JobSubmitter: number of splits:2717 > > because my each data block size is 64M,so total byte is 2717*64M/1024= 170G > > but in the summary of end i see follow info ,so the HDFS read byte is > 126792190158/1024/1024/1024 = 118G ,the two number is not very close ,why? > > File System Counters > FILE: Number of bytes read=9642910241 > FILE: Number of bytes written=120327706125 > FILE: Number of read operations=0 > FILE: Number of large read operations=0 > FILE: Number of write operations=0 > HDFS: Number of bytes read=126792190158 > HDFS: Number of bytes written=0 > HDFS: Number of read operations=8151 > HDFS: Number of large read operations=0 > HDFS: Number of write operations=0 >
Re: How can I remote debug application master
Thanks for the tips On Tue, Nov 26, 2013 at 6:17 PM, Harsh J wrote: > When constructing the AM's java command in your launcher/driver, you > could add in your remote debug JVM arguments, which should work. > > On Tue, Nov 26, 2013 at 7:34 AM, Jeff Zhang wrote: > > Hi, > > > > I build a customized application master but have some issues, is it > possible > > for me to remote debug the application master ? Thanks > > > > -- > Harsh J >
Re: How can I see the history log of non-mapreduce job in yarn
Thanks guys, just found the jira for generic history server On Wed, Nov 27, 2013 at 3:16 AM, Mayank Bansal wrote: > hi Guys, > > We don't have AHS comitted to trunk yet , We are working on branch-321 as > of now for AHS. > > Thanks, > Mayank > > > On Tue, Nov 26, 2013 at 2:16 AM, Harsh J wrote: > >> Are you talking of the new Application History Server (which is >> generic) or the Job History Server (which is part of the MR project >> and only tracks/shows MR jobs)? >> >> On Tue, Nov 26, 2013 at 7:50 AM, Jeff Zhang >> wrote: >> > I have configured the history server of yarn. But it looks like it can >> only >> > help me to see the history log of mapreduce jobs. I still could not see >> the >> > logs of non-mapreduce job. How can I see the history log of >> non-mapreduce >> > job ? >> >> >> >> -- >> Harsh J >> > >
Re: Is there any design document for YARN
Yes, I have read the paper. I know the overall architecture. I am curious to know more details of each component ( class level will be great ) On Wed, Nov 27, 2013 at 5:29 AM, Kim Chew wrote: > Hello Jeff, > > Not sure if this will help you are not, but do you have a chance to read > the "YARN-paper" yet? http://www.socc2013.org/home/program > > Kim > > > On Mon, Nov 25, 2013 at 9:34 PM, Jeff Zhang wrote: > >> >> Hi , >> >> I am reading the yarn code, so wondering whether there's any design >> document for the yarn. I found the blog post on hortonworks is very useful. >> But more details document would be helpful. Thanks >> >> >
Re: why my terasort job become a local job?
Do you set to use yarn framework in mapred-site.xml as following ? mapreduce.framework.name yarn On Tue, Nov 26, 2013 at 1:27 PM, ch huang wrote: > hi,maillist: > i run terasort in my hadoop cluster,and it run as a local job,i > do not know why ,anyone can help? > > i use hadoop version is CDH4.4 > > # sudo -u hdfs hadoop jar > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar > teragen 1000 /alex/terasort/10G-input > .. > > 13/11/26 11:57:39 INFO mapred.Task: > Task:attempt_local321416814_0001_m_00_0 is done. And is in the process > of commiting > 13/11/26 11:57:39 INFO mapred.LocalJobRunner: > 13/11/26 11:57:39 INFO mapred.Task: Task > attempt_local321416814_0001_m_00_0 is allowed to commit now > 13/11/26 11:57:39 INFO mapred.FileOutputCommitter: Saved output of task > 'attempt_local321416814_0001_m_00_0' to > hdfs://product/alex/terasort/10G-input > 13/11/26 11:57:39 INFO mapred.LocalJobRunner: > 13/11/26 11:57:39 INFO mapred.Task: Task > 'attempt_local321416814_0001_m_00_0' done. > 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Finishing task: > attempt_local321416814_0001_m_00_0 > 13/11/26 11:57:39 INFO mapred.LocalJobRunner: Map task executor complete. > 13/11/26 11:57:40 INFO mapred.JobClient: map 100% reduce 0% > 13/11/26 11:57:40 INFO mapred.JobClient: Job complete: > job_local321416814_0001 > 13/11/26 11:57:40 INFO mapred.JobClient: Counters: 19 >
Is there any design document for YARN
Hi , I am reading the yarn code, so wondering whether there's any design document for the yarn. I found the blog post on hortonworks is very useful. But more details document would be helpful. Thanks
How can I see the history log of non-mapreduce job in yarn
I have configured the history server of yarn. But it looks like it can only help me to see the history log of mapreduce jobs. I still could not see the logs of non-mapreduce job. How can I see the history log of non-mapreduce job ?
How can I remote debug application master
Hi, I build a customized application master but have some issues, is it possible for me to remote debug the application master ? Thanks
Re: How to connect to hadoop through ssh tunnel and kerberos authentication
Yes, I have the entry for CORP.EBAY.COM
here's krb5.conf
[libdefaults]
noaddresses = true
default_realm = CORP.EBAY.COM
ticket_lifetime = 36000
renew_lifetime = 604800
default_tgs_enctypes = aes256-cts aes128-cts arcfour-hmac-md5 des-cbc-md5
des-cbc-crc
default_tkt_enctypes = aes256-cts aes128-cts arcfour-hmac-md5 des-cbc-md5
des-cbc-crc
permitted_enctypes = aes256-cts aes128-cts arcfour-hmac-md5 des-cbc-md5
des-cbc-crc
dns_lookup_realm = true
dns_lookup_kdc = true
passwd_check_s_address = false
udp_preference_limit = 1
ccache_type = 3
kdc_timesync = 0
[domain_realm]
dvd-entdc-002.corp.ebay.com = CORP.EBAY.COM
dvd-entdc-001.corp.ebay.com = CORP.EBAY.COM
rhv-dmzdc-002.corp.ebay.com = CORP.EBAY.COM
.corp.ebay.com = CORP.EBAY.COM
.phx.ebay.com = CORP.EBAY.COM
corp.ebay.com = CORP.EBAY.COM
phx.ebay.com = CORP.EBAY.COM
phxaishdc9en09.corp.ebay.com = CORP.EBAY.COM
rhv-dmzdc-001.corp.ebay.com = CORP.EBAY.COM
rhv-dmzdc-003.corp.ebay.com = CORP.EBAY.COM
[realms]
CORP.EBAY.COM = {
kdc = dvd-entdc-001.corp.ebay.com:88
master_kdc = dvd-entdc-001.corp.ebay.com:88
kpasswd = dvd-entdc-001.corp.ebay.com:464
kpasswd_server = dvd-entdc-001.corp.ebay.com:464
kdc = dvd-entdc-002.corp.ebay.com:88
master_kdc = dvd-entdc-002.corp.ebay.com:88
kpasswd = dvd-entdc-002.corp.ebay.com:464
kpasswd_server = dvd-entdc-002.corp.ebay.com:464
kdc = rhv-dmzdc-001.corp.ebay.com:88
master_kdc = rhv-dmzdc-001.corp.ebay.com:88
kpasswd = rhv-dmzdc-001.corp.ebay.com:464
kpasswd_server = rhv-dmzdc-001.corp.ebay.com:464
kdc = rhv-dmzdc-002.corp.ebay.com:88
master_kdc = rhv-dmzdc-002.corp.ebay.com:88
kpasswd = rhv-dmzdc-002.corp.ebay.com:464
kpasswd_server = rhv-dmzdc-002.corp.ebay.com:464
kdc = rhv-dmzdc-003.corp.ebay.com:88
master_kdc = rhv-dmzdc-003.corp.ebay.com:88
kpasswd = rhv-dmzdc-003.corp.ebay.com:464
kpasswd_server = rhv-dmzdc-003.corp.ebay.com:464
}
On Fri, Apr 26, 2013 at 3:34 AM, Daryn Sharp wrote:
> The important part of the error is "Cannot get kdc for realm
> CORP.EBAY.COM". Check if the gateway's /etc/krb5.conf has an entry for
> CORP.EBAY.COM in the [realms] section. Or if you actually have
> appropriate dns service records for kerberos, you can use "dns_lookup_kdc =
> true".
>
> Daryn
>
>
> On Apr 25, 2013, at 12:36 AM, Jeff Zhang wrote:
>
> Hi all,
>
>
> I could connect to hadoop cluster by ssh tunnel before when there's no
> kerberos authentication. Now our cluster need to upgrade to kerberos
> authentication. I try to connect to it by ssh tunnel again. But failed.
>
> Could anyone guide me to do that ? Is there any tutorial for this ?
>
> Here's what I did.
>
>1. create a forwardable ticket in my client machine.
>2.
>
>edit ~/.ssh/config file
>
>GSSAPIAuthentication yes
>
>GSSAPIDelegateCredentials yes
>3.
>
>execute command "ssh -N -D 3600 gateway_host " to create a ssh
>connection to my gateway host
>4. config my core-site.xml file for ssh tunnel connection
>
>
> hadoophack.tunnel.port
> 3600
>
> If users connect through a SOCKS proxy, we don't
> want their SocketFactory settings interfering with the socket
> factory associated with the actual daemons.
> hadoop.rpc.socket.factory.class.default
> org.apache.hadoop.net.SocksSocketFactory
> true
>
> And there's the error message when I run "hadoop fs -ls /"
> 13/04/24 22:31:13 ERROR security.UserGroupInformation:
> PriviledgedActionException
> as:jianfezh...@corp.ebay.COMcause:javax.security.sasl.SaslException: GSS
> initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Cannot get
> kdc for realm CORP.EBAY.COM)]
> 13/04/24 22:31:13 INFO security.UserGroupInformation: Initiating logout
> for jianfezh...@corp.ebay.com
> 13/04/24 22:31:13 INFO security.UserGroupInformation: Initiating re-login
> for jianfezh...@corp.ebay.com
> 13/04/24 22:31:17 ERROR security.UserGroupInformation:
> PriviledgedActionException
> as:jianfezh...@corp.ebay.COMcause:javax.security.sasl.SaslException: GSS
> initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Cannot get
> kdc for realm CORP.EBAY.COM)]
> 13/04/24 22:31:17 WARN security.UserGroupInformation: Not attempting to
> re-login since the last re-login was attempted less than 600 seconds before.
> 13/04/24 22:31:21 ERROR security.UserGroupInformation:
> PriviledgedActionException
> as:jianfezh...@corp.ebay.COMcause:javax.security.sasl.SaslException: GSS
> initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Cannot get
> kdc for realm CORP.EBAY.COM)]
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>
--
Best Regards
Jeff Zhang
How to connect to hadoop through ssh tunnel and kerberos authentication
Hi all, I could connect to hadoop cluster by ssh tunnel before when there's no kerberos authentication. Now our cluster need to upgrade to kerberos authentication. I try to connect to it by ssh tunnel again. But failed. Could anyone guide me to do that ? Is there any tutorial for this ? Here's what I did. 1. create a forwardable ticket in my client machine. 2. edit ~/.ssh/config file GSSAPIAuthentication yes GSSAPIDelegateCredentials yes 3. execute command "ssh -N -D 3600 gateway_host " to create a ssh connection to my gateway host 4. config my core-site.xml file for ssh tunnel connection hadoophack.tunnel.port 3600 If users connect through a SOCKS proxy, we don't want their SocketFactory settings interfering with the socket factory associated with the actual daemons. hadoop.rpc.socket.factory.class.default org.apache.hadoop.net.SocksSocketFactory true And there's the error message when I run "hadoop fs -ls /" 13/04/24 22:31:13 ERROR security.UserGroupInformation: PriviledgedActionException as:jianfezh...@corp.ebay.COMcause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Cannot get kdc for realm CORP.EBAY.COM)] 13/04/24 22:31:13 INFO security.UserGroupInformation: Initiating logout for jianfezh...@corp.ebay.com 13/04/24 22:31:13 INFO security.UserGroupInformation: Initiating re-login for jianfezh...@corp.ebay.com 13/04/24 22:31:17 ERROR security.UserGroupInformation: PriviledgedActionException as:jianfezh...@corp.ebay.COMcause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Cannot get kdc for realm CORP.EBAY.COM)] 13/04/24 22:31:17 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before. 13/04/24 22:31:21 ERROR security.UserGroupInformation: PriviledgedActionException as:jianfezh...@corp.ebay.COMcause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Cannot get kdc for realm CORP.EBAY.COM)] -- Best Regards Jeff Zhang
Re: question about hadoop in maven repository
It looks like that you are using hadoop 1.x which include several sub-projects. I'm not sure Mahout support 1.x Suggest you use hadoop 0.20.x On Mon, Dec 17, 2012 at 1:31 PM, Yunming Zhang wrote: > Hi, > > I am modifying the dependencies for Mahout package, (the open source > machine learning package built on top of Hadoop), > > I am a bit confused over why there are so many hadoop dependencies in the > maven project, there are four artifactIds > 1) hadoop-core, 2) hadoop-common, 3)hadoop-mapreduce-client-**core, > 4)hadoop-mapreduce-client-**common > > I am trying to replace the hadoop jar file used to compile with my > customized version, > > Thanks > > Yunming > -- Best Regards Jeff Zhang
Re: Input splits for sequence file input
method createRecordReader will handle the record boundary issue. You can check the code for details On Mon, Dec 3, 2012 at 6:03 AM, Jeff LI wrote: > Hello, > > I was reading on the relationship between input splits and HDFS blocks and > a question came up to me: > > If a logical record crosses HDFS block boundary, let's say block#1 and > block#2, does the mapper assigned with this input split asks for (1) both > blocks, or (2) block#1 and just the part of block#2 that this logical > record extends to, or (3) block#1 and part of block#2 up to some sync point > that covers this particular logical record? Note the input is sequence > file. > > I guess my question really is: does Hadoop operate on a block basis or > does it respect some sort of logical structure within a block when it's > trying to feed the mappers with input data. > > Cheers > > Jeff > > -- Best Regards Jeff Zhang
