Re: Problem when configure the security in hadoop

?Not sure about HDFS special setup, but in general, to use HTTPs, you should have your keystore/truststore generated and config ssl-client.xml and ssl-server.xml properly. - Zhijie From: Colin Ma colin...@gmail.com Sent: Friday, July 03, 2015 1:37 AM To:

Re: ResourceManager crashes with the unnoticeable error.

?That's just some warnings from web component. It should do harm to your RM. You should check the RM log. Check if you defined HADOOP_YARN_HOME/logs or YARN_LOG_DIR?, where the daemon log lives. Thanks, Zhijie From: xeonmailinglist-gmail

Re: accessing hadoop job history

?output means the path to the history file of the job you want to view on hdfs. Thanks, Zhijie From: mehdi benchoufi benchoufi.me...@gmail.com Sent: Saturday, June 20, 2015 12:09 PM To: user@hadoop.apache.org Subject: accessing hadoop job history Hi, I ma new

Re: Using UserGroupInformation in multithread process

Do you mean TGT tickets or tokens? Anyway, they should be across threads. Did you check if you're using the same UGI object in different threads? Thanks, Zhijie From: Gaurav Gupta gaurav.gopi...@gmail.com Sent: Thursday, June 18, 2015 11:35 PM To:

Re: Lost mapreduce applications displayed in UI

?Maybe you have hit the completed app limit (1 by default). Once the limit hits, the oldest completed app will be removed from cache. - Zhijie From: hitarth trivedi t.hita...@gmail.com Sent: Tuesday, May 12, 2015 3:32 PM To: user@hadoop.apache.org Subject:

Re: Is the application tracking URL changed expected ?

?This is a known bug, but I thought it has been fixed by YARN-2246. Thanks, Zhijie From: Jeff Zhang zjf...@gmail.com Sent: Wednesday, May 06, 2015 6:23 AM To: user@hadoop.apache.org Subject: Is the application tracking URL changed expected ? I run the

Re: Yarn timeline server issue

?Hi Udit, Do you have full exception stack? Perhaps you have run into the same issue: YARN-3393. ? Thanks, Zhijie From: Udit Mehta ume...@groupon.com Sent: Thursday, April 23, 2015 12:08 PM To: user@hadoop.apache.org Subject: Yarn timeline server issue Hi,

Re: Any issue with large concurrency due to single active instance of YARN Resource Manager?

threads, where each thread is waiting for one job to finish. The memory part will probably be ok. For the 1000 threads, will there be any potential problem? Thanks, Bo -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended

Re: Any issue with large concurrency due to single active instance of YARN Resource Manager?

concurrency. To move this question further, what might be the max number of concurrent jobs which one Resource Manager could support? Is there any numbers from your experience? Thanks, Bo On Tue, Sep 2, 2014 at 12:10 AM, Zhijie Shen zs...@hortonworks.com wrote: Hi Bo, RM doesn't create

Re: Hadoop 2.5.0 unit tests failures

our suite? I also read somewhere else that there are a few flaky tests as well. Any help is appreciated. https://docs.google.com/a/qubole.com/spreadsheets/d/1bKCclEA0u9VUZvykgaRj_gBqY4xMxTBIGPJml04TXtE/edit#gid=1215903400 Thanks, Rajat -- Zhijie Shen Hortonworks Inc. http

Re: Job is reported as complete on history server while on console it shows as only half way thru

0% 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% Thanks. -- Zhijie Shen Hortonworks Inc. http

Re: ulimit for Hive

have one master node and 6 slave nodes) 5. Do I need to restart anything after configuring the ulimit? Thanks in advance, Ana -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity

Re: HDP hadoop 2.4.1 fails to run mapreduce app

) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE

Re: Hadoop 2.2.0 : Job not showing on resource management web UI

/name valueorg.apache.hadoop.mapred.ShuffleHandler/value /property /configuration What other configurations are needed to show the job in the web UI ? Thanks in advance for any assistance ! Shing -- Zhijie Shen Hortonworks Inc. http

Re: Hadoop 1.2

...@gmail.com wrote: Does hadoop map reduce code compiled against 1.2 works with Yarn? dependency groupIdorg.apache.hadoop/groupId artifactId*hadoop*-core/artifactId version1.2.1/version /dependency -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE

Re: Enable PseudoAuthenticator | org.apache.hadoop.security.authentication.client.PseudoAuthenticator

anything in the authentication. I am trying to connect oozie and hadoop, but it fails so I was thinking that I can make it use the PseudoAuthenticator somehow. I couldn't find anything in the documentation how does one sets the security authenticator. Regards, Peter -- Zhijie Shen

Re: Differences between HistoryServer and Yarn TimeLine server?

, Zhijie Shen zs...@hortonworks.comwrote: Ashwin, YARN-321 focuses on the issue in the scope of generic application history service, while YARN-1530 covers the framework specific data service. And yes, the timeline server is going to cover both. We've not such a Jira before, but it is described

Re: Differences between HistoryServer and Yarn TimeLine server?

to be YARN321+YARN1530 ? You mentioned that MR is going to integrated with timeline server,is there a jira I can watch ? Thanks, Ashwin On Wed, Apr 23, 2014 at 10:15 PM, Zhijie Shen zs...@hortonworks.comwrote: Sam, You're right. We can definitely integrate MapReduce to use the timeline

Re: Differences between HistoryServer and Yarn TimeLine server?

Server is not necessary any more.* If that's the case, why hadoop still include Job History Server?* 2014-04-23 12:56 GMT+08:00 Zhijie Shen zs...@hortonworks.com: In Hadoop 2.4, we have delivered the timeline server at a preview stage, which actually can serve some generic YARN application

Re: Differences between HistoryServer and Yarn TimeLine server?

serves the historic information of MapReduce jobs only. On Tue, Apr 22, 2014 at 8:44 PM, sam liu samliuhad...@gmail.com wrote: Hi Experts, I am confusing on these two concepts. Could you help explain the differences? Thanks! -- Zhijie Shen Hortonworks Inc. http://hortonworks.com

Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop

need to make some changes to it to run in then I need to use the new mapreduce* API and generate the new binaries? Thanks, -RR -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual

Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop

need to make some changes to it to run in then I need to use the new mapreduce* API and generate the new binaries? Thanks, -RR -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual

Re: download hadoop-2.4

-2.4.0/ On Fri, Apr 11, 2014 at 10:23 AM, lei liu liulei...@gmail.com wrote: Hadoop-2.4 is release, where can I download the hadoop-2.4 code from? Thanks, LiuLei -- Cheers -MJ -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message

Re: Container states trantition questions

nameyarn.application.classpath/name value$HADOOP_COMMON_HOME/conf,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*/value /property Appreciate for any inputs. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use

Re: hadoop version

Run hadoop version On Mon, Mar 31, 2014 at 2:22 AM, Avinash Kujur avin...@gmail.com wrote: hi, how can i know my hadoop version which i have build in my system (apart from the version which was in-built in cloudera.) regards, Avinash -- Zhijie Shen Hortonworks Inc. http

Re: task is still running on node has no disk space

will not be launched on this node. Did I miss anything? -- --Anfernee -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information

Re: Can a YARN Cient or Application Master determine when log aggregation has completed?

to determine if log aggregation is complete? Thanks, Geoff -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential

Re: how to import the hadoop code into eclipse.

showing red exclamation mark on some of the modules while i am importing it. help me out. thanks in advance. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed

Re: how to import the hadoop code into eclipse.

that. but i have some doubt while importing code. because its showing some warning and error on imported modules. i was wondering if u could give me any proper procedure link. On Thu, Mar 6, 2014 at 9:21 PM, Zhijie Shen zs...@hortonworks.com wrote: mvn eclipse:eclipse, and then import

Re: Meaning of messages in log and debugging

rule to debug a Hadoop MR application? - My application is very memory intense... is there any way to profile the memory consumption of a single container? Thanks! Best regards Yves -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message

Re: Newbie, any tutorial for install hadoop 2.3 with proper linux version

may add more PCs into it. Is there any tutorial I can learn from, such as the which linux version I should use, how to setup the linux, and how to install the hadoop step by step. I am trying to setup cluster and aim to store TB data. Any suggestion。 With Best Regards, Alex -- Zhijie

Re: YARN - Running Client with third party jars

) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 8 more How to solve this issue ? Thanks, Anand -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use

Re: query

: hello !! i want to get a information about hadoop development..from where i can get actual procedure to solve the issues.. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity

Re: Unmanaged application issue on YARN

not sure why here use the token, could any help explain that and guide me how to resolve my issue? Thanks Jeff Zhang -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which

Re: Hadoop on windows

on windows because they can run .net hadoop streaming map reduce jobs. Any way to run .net map reduce jobs on linux? That's another option we are looking at. Thanks -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended

Re: No job shown in Hadoop resource manager web UI when running jobs in the cluster

) at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1237) and on hadoop webapp like jobhistory ( I turn on the jobhistory server). It shows no job was running and no job finishing although I was running jobs. Please help me to solve this problem. Thanks!! Richard Chen -- Zhijie Shen Hortonworks Inc

Re: Aggregation service start

Please set property nameyarn.log-aggregation-enable/name valuetrue/value /property in yarn-site.xml to enable log aggregation. -Zhijie On Feb 16, 2014 6:15 PM, EdwardKing zhan...@neusoft.com wrote: hadoop 2.2.0, I want to view Tracking UI,so I visit http://172.11.12.6:8088/cluster, then I

Re: Aggregation service start

to correct? Thanks in advance. - Original Message - From: Zhijie Shen To: user@hadoop.apache.org Sent: Monday, February 17, 2014 11:11 AM Subject: Re: Aggregation service start Please set property nameyarn.log-aggregation-enable/name valuetrue/value /property in yarn

Re: job submission between 2 YARN clusters

to the wrong RM when NM is launching the container. Please comment. BTW, I just tweak the standalone app so that it will load the second cluster's configuration(yarn-site.xml) before submit the second job, it seems working. Thanks On Thu, Feb 13, 2014 at 4:28 PM, Zhijie Shen zs

Re: job submission between 2 YARN clusters

yarn.resourcemanager.scheduler.address. How that happens? I just want to make sure there's no such issue in real deployment. -- --Anfernee -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual

Re: Compression codec com.hadoop.compression.lzo.LzoCodec not found

) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:109) -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged