Re: Yarn mapreduce Logging : syslog vs stderr log files

LOG.info(“text”) —> syslog > On Mar 20, 2018, at 9:02 PM, chandan prakash > wrote: > > Hi All, > Currently my yarn MR job is writing logs to syslog and stderr. > I want to know : > how it is decided which log will go to syslog and which will go to stderr ? > Can I

Re: How to print values in console while running MapReduce application

Hi, The easiest way is to open a new window and display the log file as follow tail -f /path/to/log/file.log Best, Sultan > On Oct 4, 2017, at 5:20 PM, Tanvir Rahman wrote: > > Hello, > I have a small cluster and I am running MapReduce WordCount application in > it.

Re: Physical memory (bytes) snapshot counter question - how to get maximum memory used in reduce task

Hi Nico, Did you check the jhist file? It has all details about each task. Best, Sultan > On Apr 5, 2017, at 9:15 PM, Nico Pappagianis > wrote: > > Hi all > > I've made some memory optimizations on the reduce task and I would like to > compare the old

Re: Replacement of Hadoop-ec2 script

Check out this link https://blog.insightdatascience.com/spinning-up-a-free-hadoop-cluster-step-by-step-c406d56bae42#.9n2u8myxt On Tue, Feb 21, 2017 at 10:14 AM, Shiyuan wrote: > Hi Hadoop Users, > > The script to setup Hadoop on EC2 as described in https://wiki.apache.org/

Re: Multiple config files after building hadoop src code, Which one to modify?

After compiling the source code, you can find them here $HADOOP_HOME/etc/hadoop/ > On Feb 13, 2017, at 4:17 PM, Tanvir Rahman wrote: > > Hello everyone, > I am currently working on a research project where i need to understand the > yarn mapreduce Application Master

Re: Heartbeat between RM and AM

anks > Sunil > > > On Sat, Dec 31, 2016 at 8:20 AM Sultan Alamro <sultan.ala...@gmail.com> > wrote: > >> Hi all, >> >> Can any one tell me how I can modify the heartbeat between the RM and AM? >> I need to add new requests to the AM from the RM. >>

Heartbeat between RM and AM

Hi all, Can any one tell me how I can modify the heartbeat between the RM and AM? I need to add new requests to the AM from the RM. These requests basically are values calculated by the RM to be used by the AM online. Thanks, Sultan

Re: Configuration per job

You might need to look at the scheduler's configuration. > On Oct 25, 2016, at 10:06 PM, Jeff Zhang wrote: > > No, this is RM configuration, which is applied to all jobs. > > > > 정현진 于2016年10月26日周三 上午7:23写道： >> Hi. >> >> Is it possible to change

Map Task Execution Time

Hi there, By looking the .jhist file of a job, I see that there are startTime and finishTime for each map task. My question is, does reading the input data (local or remote) included in the execution time? Thanks, Sultan

Re: how to use Yarn API to find task/attempt status

You still can see the tasks status through the web interfaces. Look at the end of this page https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/ClusterSetup.html > On Mar 10, 2016, at 12:58 AM, Frank Luo wrote: > > Let’s say there are 10 standard M/R

Intermediate Data Spill in Mapreduce (Buffer Memory)

Hi there, I run a word count job in Hadoop 2.6.0 and I see that there are several spills for the map output. I have the following configuration: mapreduce.task.io.sort.mb = 100 mapreduce.map.sort.spill.percent = 0.80 After running the job, the Map output bytes = 222660096. By looking at the

Blocks processed by which task?

Hi there, How do I know which block in my HDFS processed by which task? I want to make sure if my Hadoop applies "Locality" concept or not. Thanks,

Re: Running multiple copies of each task

e looking for mapreduce.reduce.speculative > Be careful, for some reason, this fell into my spam folder. > > Regards, > LLoyd > > On 3 December 2015 at 01:05, Sultan Alamro <sultan.ala...@gmail.com> > wrote: > > Hi there, > > > > I have been lookin

Running multiple copies of each task

Hi there, I have been looking at the hadoop source code 2.6.0 trying to understand the low level details and how the framework is actually working. I have a simple idea and I am trying to figure out where and how the idea can be implemented. The idea can be described in one sentence: "Running