Re: How to implement a servlet to submit job to Hadoop 2.6.0 cluster

Thanks you were right but loading *-site.xml files in hdfs for example in a folder /configurationFiles/ created by: hdfs dfs -mkdir /configurationFiles and writing config.addResource(new Path("hdfs://master:9000/configurationFiles/core-site.xml")); i get the same error... this is the complete

Can't see logs of a finished job.

Hi, I have configured the proxy server and the historyserver in Yarn Mapreduce, and they are running properly [1]. But, I can’t see the logs of an executed job [2]. I even have set history params in |yarn-site.xml| [3]. Am I missing some configuration to see the logs of a finished job? [1]

Hive update functionality for External tables

Hi All, With the version 0.14 Hive supports Update functionality. I have tried updating a internal table and it works just like RDBMS Update command( though it takes more time to update). Please let me know if it is possible to UPDATE the EXTERNAL TABLES IN HIVE . Eagerly waiting for the respon

Re: Can't see logs of a finished job.

In the |mapred-site.xml| I have also this configuration [4]: [4] |mapreduce.jobhistory.address hadoop-coc-1:10020 mapreduce.jobhistory.webapp.address hadoop-coc-1:19888 mapreduce.jobhistory.done-dir /home/xubuntu/Programs/hadoop/logs/history/done mapreduce.jobhistory.intermed

can't submit a job

Hi, I am trying to launch a job that I have configured in in java, but I get an error related to the containers [1]. I don’t understand why I can’t submit the a job. Why get this error? What can I do to fix it? Thanks, [1] Log of |logs/yarn-xubuntu-nodemanager-hadoop-coc-1.log| |15/05/28 09

Re: can't submit a job

Have you checked the link http://192.168.56.101:9046/proxy/application_1432817967879_0003 ? You should get come clue from logs of the 2 attempts. On Thu, May 28, 2015 at 6:42 AM, xeonmailinglist-gmail < xeonmailingl...@gmail.c

Fwd: Re: can't submit a job

The error that I got is [1], but I still don’t understand why I get this error. I couldn’t find more detail about it. Any suggestion? [1] | Application application_1432817967879_0003 failed 2 times due to AM Container for appattempt_1432817967879_0003_02 exited with exitCode: 1 For more de

Re: Fwd: Re: can't submit a job

If I click in the logs link, I get the error. So it seems that my job isn't launching and the container aren't working. I need to fix both errors. Any suggestion? ``` Failed redirect for container_1432817967879_0003_01_01 ResourceManager * RM Home

Re: Fwd: Re: can't submit a job

I have found why I couldn’t access the container. I needed to have in |yarn-site.xml| the property: | yarn.log.server.url http://192.168.56.101:19888/jobhistory/logs/ | On 05/28/2015 04:01 PM, xeonmailinglist-gmail wrote: If I click in the logs link, I get the error. So it seems that my job

Apache Hadoop tests fail with UnknownHostException

Hi experts, I tried to build apache hadoop mapreduce project on my Linux host, but got some test failures on hadoop-mapreduce-client-jobclient subproject. Most of these test errors are caused by UnknownHostException. Following is one of the stacktrace: --

One job blocks all queues.

Hello, I am experimenting the use of multi-tenancy in Hadoop. I have a Hive queries which does never give a result and whose containers seem to freeze forever. It is basically a join where all key values of both input tables are the same. I understand there can be bugs in Hive and they will be c

RE: One job blocks all queues.

How are your queues configured ? -Original Message- From: "Julien Carme" Sent: ‎5/‎28/‎2015 11:29 AM To: "user@hadoop.apache.org" Subject: One job blocks all queues. Hello, I am experimenting the use of multi-tenancy in Hadoop. I have a Hive queries which does never give a result a

Re: One job blocks all queues.

Julien - Sounds like you are using default Capacity Scheduler settings which has minimum-user-limit-percent = 100, meaning the minimum guaranteed resources for a single user is 100% Read more about this property here - http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_system-admin-gu

Re: One job blocks all queues.

Thanks for your answers. My understanding is that minimum-user-limit-percent handles resource sharing inside a queue. My problem is that a single user blocks its queue (which is OK), but all other queues as well (which is not OK). My configuration is: yarn.scheduler.capacity.root.accessible-node

Re: Can't see logs of a finished job.

Hey, the job is finished. In order to see the logs for the finished job from webui, you need add one more configuration: I see that you have already started jobhistoryserver (Assume the jhs address is localhost:19888). Add this configuration to yarn-site.xml: yarn.log.server.url h

Question about avoiding reflection with isolated class loader

Hi, I have a question that is probably related to MAPREDUCE-1700 / 5751 / 5957 . Basically I have also encountered the issue that with separate classloaders while developing a programming framework where I have to use reflection inside the applicat

Re: DataNode Timeout exceptions.

Hi Ted , I have only 3 Datanodes. When I check the logs , I see the following exception in the DataNode log and no exceptions in the NameNode log. Stack Trace from the DataNode log. 2015-05-27 10:52:34,741 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: exception: java.net.SocketTimeoutExc

ResouceManager hung： org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 1000

Hi, I met the same problem as : http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201303.mbox/%3c482c5f6f-6feb-4552-99f5-07c8b54ac...@apache.org%3E Any idea

Re: ResouceManager hung： org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 1000

Hi Can you take thread dump and verify it? jstack > RM.out OR kill -3 (Note : head dump will be logged in out file) Thanks & Regards Rohith Sharma K S > On May 29, 2015, at 8:43 AM, jason lu wrote: > > > Hi, > I met the same problem as : > http://mail-archives.apache.org/mod_mbox/ha

Re: ResouceManager hung： org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 1000

I forgot to do that before restart the process. > 在 2015年5月29日，11:17，Rohith Sharma 写道： > > Hi > > Can you take thread dump and verify it? > > jstack > RM.out > OR > kill -3 (Note : head dump will be logged in out file) > > Thanks & Regards > Rohith Sharma K S > >> On May 29, 2015, at 8:4

Cluster sizing

Hi, I wanted to know how I can determine how many nodes with cores/storage in TB and RAM needed, if I will receieve the data volume increase from 1TB to 100TB per day. Can someone help me here to create a excel based on this. Thanks

Re: Cluster sizing

Can you share some more inputs on requirement . What is the analytics usecase ? ( Batch Processing , Real Time , In-Memory Requirements ) Which distribution of Hadoop ? What is the storage growth rate ? What are the data ingest requirements ? What kind of jobs will run on the cluster ? What is t