v2.8.0: Setting PID dir EnvVars: libexec/thing-config.sh or etc/hadoop/thing-env.sh ?

[Kevin Buckley]

I've noted in

https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/ClusterSetup.html

that it says

8<8<8<8<8<
See etc/hadoop/hadoop-env.sh for other examples.

Other useful configuration parameters that you can customize include:

HADOOP_PID_DIR - The directory where the daemons’ process id files
 are stored.

HADOOP_LOG_DIR - The directory where the daemons’ log files are
 stored. Log files are automatically created if
 they don’t exist.

HADOOP_HEAPSIZE / YARN_HEAPSIZE - The maximum amount of heapsize
 to use, in MB e.g. if the varibale is set to 1000
 the heap will be set to 1000MB. This is used to
 configure the heap size for the daemon. By
 default, the value is 1000. If you want to
 configure the values separately for each deamon
 you can use.

In most cases, you should specify the HADOOP_PID_DIR and HADOOP_LOG_DIR
directories such that they can only be written to by the users that
are going to run the hadoop daemons. Otherwise there is the potential
for a symlink attack.
8<8<8<8<8<

and I have recently had a need to move my PID-files from their default,
/tmp, location, so as to avoid an over-aggressive /tmp cleanup.


So, if I am in

 /path/to/hadoop-2.8.0/etc/hadoop

I can see

$ grep PID *
hadoop-env.cmd:set HADOOP_PID_DIR=%HADOOP_PID_DIR%
hadoop-env.cmd:set HADOOP_SECURE_DN_PID_DIR=%HADOOP_PID_DIR%
hadoop-env.sh:export HADOOP_PID_DIR=${HADOOP_PID_DIR}
hadoop-env.sh:export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
mapred-env.sh:#export HADOOP_MAPRED_PID_DIR= # The pid files are
stored. /tmp by default.
$

so clearly PID_DIR values could be overriden there, although I note that the

 yarn-env.sh

file, unlike its "hadoop" and "mapred" bretheren, doesn't contain
any PID_DIR default lines, nor even commented suggestion lines for

 YARN_PID_DIR

which is used in, for exmaple,

 hadoop-2.8.0/sbin/yarn-daemon.sh



However, I've also noted that many of the Hadoop sbin scripts
will source a related file from

 hadoop-2.8.0/libexec/

so for example, a desired value for the env var

 YARN_PID_DIR

might thus be set in either (or both ?) of

 hadoop-2.8.0/etc/hadoop/yarn-env.sh
 hadoop-2.8.0/libexec/yarn-config.sh

and similarly,

  HADOOP_PID_DIR

might be set in either (or both ?) of

 hadoop-2.8.0/libexec/hadoop-config.sh
 hadoop-2.8.0/etc/hadoop/hadoop-env.sh


So,

is there a "to be preferred" file choice, between those two,
within which to set certain classes of EnvVars ?


Any info/pointers welcome,
Kevin

---
Kevin M. Buckley

eScience Consultant
School of Engineering and Computer Science
Victoria University of Wellington
New Zealand

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Re: Any ideas when Hadoop 3 will become final?

[Brahma Reddy Battula]

Tentative dates are mentioned in following link, it might be available mid of 
August-2017.



https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release

Hadoop 3.0.0 release - Hadoop - Apache Software 
Foundation
cwiki.apache.org
Release schedule. For Hadoop 3, we are planning to "release early, release 
often" to quickly iterate on feedback collected from downstream projects.





-Brahma Reddy Battula


From: Jasson Chenwei 
Sent: Tuesday, June 13, 2017 6:11 AM
To: Karthik Shyamsunder
Cc: user
Subject: Re: Any ideas when Hadoop 3 will become final?

Good one. I also want to know!! Also, are there any efforts to make projects 
(e.g., Hive and Spark) in Hadoop-eco to be compatible with Hadoop-3.0 ?


Thanks,

Wei

On Sat, Jun 10, 2017 at 5:58 AM, Karthik Shyamsunder 
> wrote:
Folks,

Any ideas when Hadoop 3 will become final? Its currently Hadoop 3.0.0 alpah3.  
Are we looking at 2017 or 2018 for GA?  Please advise.

Sincerely,

Karthik

Re: How to monitor YARN application memory per container?

[Shmuel Blitz]

Hi Sunil,

Thanks for your response.

Here is the response I get when running  "yarn node -status {nodeId}" :

Node Report :
Node-Id : myNode:4545
Rack : /default
Node-State : RUNNING
Node-Http-Address : muNode:8042
Last-Health-Update : Wed 14/Jun/17 08:25:43:261EST
Health-Report :
Containers : 7
Memory-Used : 44032MB
Memory-Capacity : 49152MB
CPU-Used : 16 vcores
CPU-Capacity : 48 vcores
Node-Labels :

However, this is information regarding the entire node, containing all
containers.

I have no way of using this to see the value I give to '
spark.executor.memory' makes sense or not.

I'm looking for memory usage/allocated information *per-container*.

Shmuel

On Wed, Jun 14, 2017 at 4:04 PM, Sunil G  wrote:

> Hi Shmuel
>
> In Hadoop 2.8 release line, you could check "yarn node -status {nodeId}"
> CLI command or "http:///ws/v1/cluster/nodes/{nodeid}"
> REST end point to get container's actual resource usage per node. You could
> also check the same in any of Hadoop 3.0 alpha releases as well.
>
> Thanks
> Sunil
>
> On Tue, Jun 13, 2017 at 11:29 PM Shmuel Blitz 
> wrote:
>
>> Hi,
>>
>> Thanks for your response.
>>
>> The /metrics API returns a blank page on our RM.
>>
>> The /jmx API has some metrics, but these are the same metrics we are
>> already loading into data-dog.
>> It's not good enough, because it doesn't break down the memory use by
>> container.
>>
>> I need the by-container breakdown because resource allocation is per
>> container and I would like to se if my job is really using up all the
>> allocated memory.
>>
>> Shmuel
>>
>> On Tue, Jun 13, 2017 at 6:05 PM, Sidharth Kumar <
>> sidharthkumar2...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I guess you can get it from http://:/jmx
>>> or /metrics
>>>
>>> Regards
>>> Sidharth
>>> LinkedIn: www.linkedin.com/in/sidharthkumar2792
>>>
>>> On 13-Jun-2017 6:26 PM, "Shmuel Blitz" 
>>> wrote:
>>>
 (This question has also been published on StackOveflow
 )

 I am looking for a way to monitor memory usage of YARN containers over
 time.

 Specifically - given a YARN application-id, how can you get a graph,
 showing the memory usage of each of its containers over time?

 The main goal is to better fit memory allocation requirements for our
 YARN applications (Spark / Map-Reduce), to avoid over allocation and
 cluster resource waste. A side goal would be the ability to debug memory
 issues when developing our jobs and attempting to pick reasonable resource
 allocations.

 We've tried using the Data-Dog integration, But it doesn't break down
 the metrics by container.

 Another approach was to parse the hadoop-yarn logs. These logs have
 messages like:

 Memory usage of ProcessTree 57251 for container-id
 container_e116_1495951495692_35134_01_01: 1.9 GB of 11 GB physical
 memory used; 14.4 GB of 23.1 GB virtual memory used
 Parsing the logs correctly can yield data that can be used to plot a
 graph of memory usage over time.

 That's exactly what we want, but there are two downsides:

 It involves reading human-readable log lines and parsing them into
 numeric data. We'd love to avoid that.
 If this data can be consumed otherwise, we're hoping it'll have more
 information that we might be interest in in the future. We wouldn't want to
 put the time into parsing the logs just to realize we need something else.
 Is there any other way to extract these metrics, either by plugging in
 to an existing producer or by writing a simple listener?

 Perhaps a whole other approach?

 --
 [image: Logo]
 
 Shmuel Blitz
 *Big Data Developer*
 www.similarweb.com
 

 
  Like
 Us
 

 
  Follow
 Us
 

 
  Watch
 Us
 

 

Re: How to monitor YARN application memory per container?

[Sunil G]

Hi Shmuel

In Hadoop 2.8 release line, you could check "yarn node -status {nodeId}"
CLI command or "http:///ws/v1/cluster/nodes/{nodeid}"
REST end point to get container's actual resource usage per node. You could
also check the same in any of Hadoop 3.0 alpha releases as well.

Thanks
Sunil

On Tue, Jun 13, 2017 at 11:29 PM Shmuel Blitz 
wrote:

> Hi,
>
> Thanks for your response.
>
> The /metrics API returns a blank page on our RM.
>
> The /jmx API has some metrics, but these are the same metrics we are
> already loading into data-dog.
> It's not good enough, because it doesn't break down the memory use by
> container.
>
> I need the by-container breakdown because resource allocation is per
> container and I would like to se if my job is really using up all the
> allocated memory.
>
> Shmuel
>
> On Tue, Jun 13, 2017 at 6:05 PM, Sidharth Kumar <
> sidharthkumar2...@gmail.com> wrote:
>
>> Hi,
>>
>> I guess you can get it from http://:/jmx
>> or /metrics
>>
>> Regards
>> Sidharth
>> LinkedIn: www.linkedin.com/in/sidharthkumar2792
>>
>> On 13-Jun-2017 6:26 PM, "Shmuel Blitz" 
>> wrote:
>>
>>> (This question has also been published on StackOveflow
>>> )
>>>
>>> I am looking for a way to monitor memory usage of YARN containers over
>>> time.
>>>
>>> Specifically - given a YARN application-id, how can you get a graph,
>>> showing the memory usage of each of its containers over time?
>>>
>>> The main goal is to better fit memory allocation requirements for our
>>> YARN applications (Spark / Map-Reduce), to avoid over allocation and
>>> cluster resource waste. A side goal would be the ability to debug memory
>>> issues when developing our jobs and attempting to pick reasonable resource
>>> allocations.
>>>
>>> We've tried using the Data-Dog integration, But it doesn't break down
>>> the metrics by container.
>>>
>>> Another approach was to parse the hadoop-yarn logs. These logs have
>>> messages like:
>>>
>>> Memory usage of ProcessTree 57251 for container-id
>>> container_e116_1495951495692_35134_01_01: 1.9 GB of 11 GB physical
>>> memory used; 14.4 GB of 23.1 GB virtual memory used
>>> Parsing the logs correctly can yield data that can be used to plot a
>>> graph of memory usage over time.
>>>
>>> That's exactly what we want, but there are two downsides:
>>>
>>> It involves reading human-readable log lines and parsing them into
>>> numeric data. We'd love to avoid that.
>>> If this data can be consumed otherwise, we're hoping it'll have more
>>> information that we might be interest in in the future. We wouldn't want to
>>> put the time into parsing the logs just to realize we need something else.
>>> Is there any other way to extract these metrics, either by plugging in
>>> to an existing producer or by writing a simple listener?
>>>
>>> Perhaps a whole other approach?
>>>
>>> --
>>> [image: Logo]
>>> 
>>> Shmuel Blitz
>>> *Big Data Developer*
>>> www.similarweb.com
>>> 
>>>
>>> 
>>>  Like
>>> Us
>>> 
>>>
>>> 
>>>  Follow
>>> Us
>>> 
>>>
>>> 
>>>  Watch
>>> Us
>>> 
>>>
>>> 
>>>  Read
>>> Us
>>> 
>>>
>>
>
>
> --
> [image: Logo]
> 
> Shmuel Blitz
> *Big Data Developer*
> www.similarweb.com
> 
>
> 
>  Like
> Us
> 
>
> 
>  Follow
> Us
> 
>
> 

RE: HDFS file replication to slave nodes not working

[Brahma Reddy Battula]

Yes, you can configure “dfs.datanode.data.dir”.

Reference for default configurations:
http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml


Regards
Brahma Reddy Battula

From: Bhushan Pathak [mailto:bhushan.patha...@gmail.com]
Sent: 14 June 2017 17:47
To: Brahma Reddy Battula
Cc: common-u...@hadoop.apache.org
Subject: Re: HDFS file replication to slave nodes not working

Any way I can tell hadoop to use the /mnt dir instead of 
/tmp/hadoop-{user-name} directory to store the files?

Thanks
Bhushan Pathak

Thanks
Bhushan Pathak

On Wed, Jun 14, 2017 at 3:06 PM, Brahma Reddy Battula 
> wrote:


Please see my comments inline.



Regards
Brahma Reddy Battula

From: Bhushan Pathak 
[mailto:bhushan.patha...@gmail.com]
Sent: 14 June 2017 17:14
To: common-u...@hadoop.apache.org
Subject: HDFS file replication to slave nodes not working

Hello,

I have hadoop 2.7.3 running on a 3-node cluster [1 master, 2 slaves]. The 
hdfs-site.xml file has the following config -

dfs.namenode.name.dir
file:/mnt/hadoop_store/datanode


dfs.datanode.name.dir
file:/mnt/hadoop_store/namenode


===> property should be “dfs.datanode.data.dir”. Please have a look following 
for all default configurations.
http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

I used the 'hdfs -put' command to upload 3 csv files to HDFS, which was 
successful.

My assumption is that the 3 csv files should be present on all 3 nodes, either 
under the datanode or the namenode directory. On the master, I can see the 
following files -

[hadoop@master hadoop-2.7.3]$ bin/hdfs dfs -ls /usr/hadoop
Found 3 items
-rw-r--r--   3 hadoop supergroup 124619 2017-06-14 14:34 
/usr/hadoop/Final_Album_file.csv
-rw-r--r--   3 hadoop supergroup  68742 2017-06-14 14:34 
/usr/hadoop/Final_Artist_file.csv
-rw-r--r--   3 hadoop supergroup2766110 2017-06-14 14:34 
/usr/hadoop/Final_Tracks_file.csv
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/namenode/
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/
current  in_use.lock
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/current/
edits_001-002  
edits_027-028  
edits_055-056
edits_003-004  
edits_029-030  
edits_057-058
edits_005-006  
edits_031-032  
edits_059-060
edits_007-008  
edits_033-034  
edits_061-064
edits_009-010  
edits_035-036  
edits_065-096
edits_011-012  
edits_037-038  
edits_inprogress_097
edits_013-014  
edits_039-040  fsimage_064
edits_015-016  
edits_041-042  fsimage_064.md5
edits_017-017  
edits_043-044  fsimage_096
edits_018-019  
edits_045-046  fsimage_096.md5
edits_020-020  
edits_047-048  seen_txid
edits_021-022  
edits_049-050  VERSION
edits_023-024  
edits_051-052
edits_025-026  
edits_053-054
[hadoop@master hadoop-2.7.3]$

While on the 2 slave nodes, there are only empty directories. Is my assumption 
that the 3 csv files should be replicated to slave nodes as well correct? If 
yes, why are  they missing from the slave nodes? Additionally, are the files 
that I see in datanode/current directory of master the actual csv files that I 
have uploaded?


Yes, it will replicate to 3 nodes.(it’s based on “dfs.replication” which is “3” 
by default)
The location which you are checking is wrong, since property is wrong..and by 
default it will stored under “/tmp/Hadoop-${user-name}}.
Data under “datanode/current directory” is meta data for all operations.

Please go through the following design to know more about HDFS.
http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html


Thanks
Bhushan Pathak

Re: HDFS file replication to slave nodes not working

[Bhushan Pathak]

Any way I can tell hadoop to use the /mnt dir instead of
/tmp/hadoop-{user-name} directory to store the files?

Thanks
Bhushan Pathak

Thanks
Bhushan Pathak

On Wed, Jun 14, 2017 at 3:06 PM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

>
>
>
>
> Please see my comments inline.
>
>
>
>
>
>
>
> Regards
>
> Brahma Reddy Battula
>
>
>
> *From:* Bhushan Pathak [mailto:bhushan.patha...@gmail.com]
> *Sent:* 14 June 2017 17:14
> *To:* common-u...@hadoop.apache.org
> *Subject:* HDFS file replication to slave nodes not working
>
>
>
> Hello,
>
>
>
> I have hadoop 2.7.3 running on a 3-node cluster [1 master, 2 slaves]. The
> hdfs-site.xml file has the following config -
>
> 
>
> dfs.namenode.name.dir
>
> file:/mnt/hadoop_store/datanode
>
> 
>
> 
>
> dfs.datanode.name.dir
>
> file:/mnt/hadoop_store/namenode
>
> 
>
>
>
> =è *property should be “dfs.datanode.data.dir”. Please have a look
> following for all default configurations.*
>
> *http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> *
>
>
>
> I used the 'hdfs -put' command to upload 3 csv files to HDFS, which was
> successful.
>
>
>
> My assumption is that the 3 csv files should be present on all 3 nodes,
> either under the datanode or the namenode directory. On the master, I can
> see the following files -
>
>
>
> [hadoop@master hadoop-2.7.3]$ bin/hdfs dfs -ls /usr/hadoop
>
> Found 3 items
>
> -rw-r--r--   3 hadoop supergroup 124619 2017-06-14 14:34
> /usr/hadoop/Final_Album_file.csv
>
> -rw-r--r--   3 hadoop supergroup  68742 2017-06-14 14:34
> /usr/hadoop/Final_Artist_file.csv
>
> -rw-r--r--   3 hadoop supergroup2766110 2017-06-14 14:34
> /usr/hadoop/Final_Tracks_file.csv
>
> [hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/namenode/
>
> [hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/
>
> current  in_use.lock
>
> [hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/current/
>
> edits_001-002  
> edits_027-028
>  edits_055-056
>
> edits_003-004  
> edits_029-030
>  edits_057-058
>
> edits_005-006  
> edits_031-032
>  edits_059-060
>
> edits_007-008  
> edits_033-034
>  edits_061-064
>
> edits_009-010  
> edits_035-036
>  edits_065-096
>
> edits_011-012  
> edits_037-038
>  edits_inprogress_097
>
> edits_013-014  
> edits_039-040
>  fsimage_064
>
> edits_015-016  
> edits_041-042
>  fsimage_064.md5
>
> edits_017-017  
> edits_043-044
>  fsimage_096
>
> edits_018-019  
> edits_045-046
>  fsimage_096.md5
>
> edits_020-020  
> edits_047-048
>  seen_txid
>
> edits_021-022  
> edits_049-050
>  VERSION
>
> edits_023-024  edits_051-
> 052
>
> edits_025-026  edits_053-
> 054
>
> [hadoop@master hadoop-2.7.3]$
>
>
>
> While on the 2 slave nodes, there are only empty directories. Is my
> assumption that the 3 csv files should be replicated to slave nodes as well
> correct? If yes, why are  they missing from the slave nodes? Additionally,
> are the files that I see in datanode/current directory of master the actual
> csv files that I have uploaded?
>
>
>
>
>
> *Yes, it will replicate to 3 nodes.(it’s based on *“dfs.replication” *which
> is “3” by default)*
>
> *The location which you are checking is wrong, since property is
> wrong..and by default it will stored under “/tmp/Hadoop-${user-name}}. *
>
> *Data under “datanode/current directory” is meta data for all operations.*
>
>
>
> *Please go through the following design to know more about HDFS.*
>
> *http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
> *
>
>
>
>
>
> Thanks
>
> Bhushan Pathak
>

RE: HDFS file replication to slave nodes not working

[Brahma Reddy Battula]

Please see my comments inline.



Regards
Brahma Reddy Battula

From: Bhushan Pathak [mailto:bhushan.patha...@gmail.com]
Sent: 14 June 2017 17:14
To: common-u...@hadoop.apache.org
Subject: HDFS file replication to slave nodes not working

Hello,

I have hadoop 2.7.3 running on a 3-node cluster [1 master, 2 slaves]. The 
hdfs-site.xml file has the following config -

dfs.namenode.name.dir
file:/mnt/hadoop_store/datanode


dfs.datanode.name.dir
file:/mnt/hadoop_store/namenode


===> property should be “dfs.datanode.data.dir”. Please have a look following 
for all default configurations.
http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

I used the 'hdfs -put' command to upload 3 csv files to HDFS, which was 
successful.

My assumption is that the 3 csv files should be present on all 3 nodes, either 
under the datanode or the namenode directory. On the master, I can see the 
following files -

[hadoop@master hadoop-2.7.3]$ bin/hdfs dfs -ls /usr/hadoop
Found 3 items
-rw-r--r--   3 hadoop supergroup 124619 2017-06-14 14:34 
/usr/hadoop/Final_Album_file.csv
-rw-r--r--   3 hadoop supergroup  68742 2017-06-14 14:34 
/usr/hadoop/Final_Artist_file.csv
-rw-r--r--   3 hadoop supergroup2766110 2017-06-14 14:34 
/usr/hadoop/Final_Tracks_file.csv
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/namenode/
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/
current  in_use.lock
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/current/
edits_001-002  
edits_027-028  
edits_055-056
edits_003-004  
edits_029-030  
edits_057-058
edits_005-006  
edits_031-032  
edits_059-060
edits_007-008  
edits_033-034  
edits_061-064
edits_009-010  
edits_035-036  
edits_065-096
edits_011-012  
edits_037-038  
edits_inprogress_097
edits_013-014  
edits_039-040  fsimage_064
edits_015-016  
edits_041-042  fsimage_064.md5
edits_017-017  
edits_043-044  fsimage_096
edits_018-019  
edits_045-046  fsimage_096.md5
edits_020-020  
edits_047-048  seen_txid
edits_021-022  
edits_049-050  VERSION
edits_023-024  
edits_051-052
edits_025-026  
edits_053-054
[hadoop@master hadoop-2.7.3]$

While on the 2 slave nodes, there are only empty directories. Is my assumption 
that the 3 csv files should be replicated to slave nodes as well correct? If 
yes, why are  they missing from the slave nodes? Additionally, are the files 
that I see in datanode/current directory of master the actual csv files that I 
have uploaded?


Yes, it will replicate to 3 nodes.(it’s based on “dfs.replication” which is “3” 
by default)
The location which you are checking is wrong, since property is wrong..and by 
default it will stored under “/tmp/Hadoop-${user-name}}.
Data under “datanode/current directory” is meta data for all operations.

Please go through the following design to know more about HDFS.
http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html


Thanks
Bhushan Pathak

HDFS file replication to slave nodes not working

[Bhushan Pathak]

Hello,

I have hadoop 2.7.3 running on a 3-node cluster [1 master, 2 slaves]. The
hdfs-site.xml file has the following config -

dfs.namenode.name.dir
file:/mnt/hadoop_store/datanode


dfs.datanode.name.dir
file:/mnt/hadoop_store/namenode


I used the 'hdfs -put' command to upload 3 csv files to HDFS, which was
successful.

My assumption is that the 3 csv files should be present on all 3 nodes,
either under the datanode or the namenode directory. On the master, I can
see the following files -

[hadoop@master hadoop-2.7.3]$ bin/hdfs dfs -ls /usr/hadoop
Found 3 items
-rw-r--r--   3 hadoop supergroup 124619 2017-06-14 14:34
/usr/hadoop/Final_Album_file.csv
-rw-r--r--   3 hadoop supergroup  68742 2017-06-14 14:34
/usr/hadoop/Final_Artist_file.csv
-rw-r--r--   3 hadoop supergroup2766110 2017-06-14 14:34
/usr/hadoop/Final_Tracks_file.csv
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/namenode/
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/
current  in_use.lock
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/current/
edits_001-002
 edits_027-028
 edits_055-056
edits_003-004
 edits_029-030
 edits_057-058
edits_005-006
 edits_031-032
 edits_059-060
edits_007-008
 edits_033-034
 edits_061-064
edits_009-010
 edits_035-036
 edits_065-096
edits_011-012
 edits_037-038
 edits_inprogress_097
edits_013-014
 edits_039-040  fsimage_064
edits_015-016
 edits_041-042
 fsimage_064.md5
edits_017-017
 edits_043-044  fsimage_096
edits_018-019
 edits_045-046
 fsimage_096.md5
edits_020-020
 edits_047-048  seen_txid
edits_021-022
 edits_049-050  VERSION
edits_023-024
 edits_051-052
edits_025-026
 edits_053-054
[hadoop@master hadoop-2.7.3]$

While on the 2 slave nodes, there are only empty directories. Is my
assumption that the 3 csv files should be replicated to slave nodes as well
correct? If yes, why are  they missing from the slave nodes? Additionally,
are the files that I see in datanode/current directory of master the actual
csv files that I have uploaded?


Thanks
Bhushan Pathak

Re: Mapreduce application failed on Distributed scheduler on YARN.

[Konstantinos Karanasos]

Hi Wei,

Can you please share your yarn-site.xml and the command you used for
running wordcount?

Thanks,
Konstantinos

On Tue, Jun 13, 2017 at 11:44 Jasson Chenwei 
wrote:

> hi all
>
> I have set up a distributed scheduler using a new feature in Hadoop-3.0. My
> Hadoop version is hadoop-3.0.0-appha3. I have enabled the
> opportunistic container and distributed scheduler in
> yarn-site.xml following the guide. But wordcount application master failed
> to launch as follow:
>
> *2017-06-13 12:34:11,036 INFO [main] org.eclipse.jetty.server.Server:
> Started @5116ms*
> *2017-06-13 12:34:11,036 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
> Web app mapreduce started at 45559*
> *2017-06-13 12:34:11,039 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE
> job_1497050650910_0004*
> *2017-06-13 12:34:11,041 INFO [main]
> org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class
> java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler:
> class org.apache.hadoop.ipc.DefaultRpcScheduler*
> *2017-06-13 12:34:11,042 INFO [Socket Reader #1 for port 36026]
> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36026*
> *2017-06-13 12:34:11,045 INFO [IPC Server Responder]
> org.apache.hadoop.ipc.Server: IPC Server Responder: starting*
> *2017-06-13 12:34:11,045 INFO [IPC Server listener on 36026]
> org.apache.hadoop.ipc.Server: IPC Server listener on 36026: starting*
> *2017-06-13 12:34:11,083 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> nodeBlacklistingEnabled:true*
> *2017-06-13 12:34:11,083 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> maxTaskFailuresPerNode is 3*
> *2017-06-13 12:34:11,083 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> blacklistDisablePercent is 33*
> *2017-06-13 12:34:11,090 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the
> mappers will be scheduled using OPPORTUNISTIC containers*
> *2017-06-13 12:34:11,132 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
> Connecting to ResourceManager at localhost/127.0.0.1:8030
> *
> *2017-06-13 12:34:11,193 WARN [main] org.apache.hadoop.ipc.Client:
> Exception encountered while connecting to the server :
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1497050650910_0004_02*
> *2017-06-13 12:34:11,203 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while
> registering*
> *org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1497050650910_0004_02*
> * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)*
> * at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)*
> * at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)*
> * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
> * at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)*
> * at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)*
> * at
>
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)*
> * at
>
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)*
> * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> * at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*
> * at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
> * at java.lang.reflect.Method.invoke(Method.java:498)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)*
> * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> 

Re: How to monitor YARN application memory per container?

[Shmuel Blitz]

Hi,

Thanks for your response.

The /metrics API returns a blank page on our RM.

The /jmx API has some metrics, but these are the same metrics we are
already loading into data-dog.
It's not good enough, because it doesn't break down the memory use by
container.

I need the by-container breakdown because resource allocation is per
container and I would like to se if my job is really using up all the
allocated memory.

Shmuel

On Tue, Jun 13, 2017 at 6:05 PM, Sidharth Kumar  wrote:

> Hi,
>
> I guess you can get it from http://:/jmx
> or /metrics
>
> Regards
> Sidharth
> LinkedIn: www.linkedin.com/in/sidharthkumar2792
>
> On 13-Jun-2017 6:26 PM, "Shmuel Blitz" 
> wrote:
>
>> (This question has also been published on StackOveflow
>> )
>>
>> I am looking for a way to monitor memory usage of YARN containers over
>> time.
>>
>> Specifically - given a YARN application-id, how can you get a graph,
>> showing the memory usage of each of its containers over time?
>>
>> The main goal is to better fit memory allocation requirements for our
>> YARN applications (Spark / Map-Reduce), to avoid over allocation and
>> cluster resource waste. A side goal would be the ability to debug memory
>> issues when developing our jobs and attempting to pick reasonable resource
>> allocations.
>>
>> We've tried using the Data-Dog integration, But it doesn't break down the
>> metrics by container.
>>
>> Another approach was to parse the hadoop-yarn logs. These logs have
>> messages like:
>>
>> Memory usage of ProcessTree 57251 for container-id
>> container_e116_1495951495692_35134_01_01: 1.9 GB of 11 GB physical
>> memory used; 14.4 GB of 23.1 GB virtual memory used
>> Parsing the logs correctly can yield data that can be used to plot a
>> graph of memory usage over time.
>>
>> That's exactly what we want, but there are two downsides:
>>
>> It involves reading human-readable log lines and parsing them into
>> numeric data. We'd love to avoid that.
>> If this data can be consumed otherwise, we're hoping it'll have more
>> information that we might be interest in in the future. We wouldn't want to
>> put the time into parsing the logs just to realize we need something else.
>> Is there any other way to extract these metrics, either by plugging in to
>> an existing producer or by writing a simple listener?
>>
>> Perhaps a whole other approach?
>>
>> --
>> [image: Logo]
>> 
>> Shmuel Blitz
>> *Big Data Developer*
>> www.similarweb.com
>> 
>>
>> 
>>  Like
>> Us
>> 
>>
>> 
>>  Follow
>> Us
>> 
>>
>> 
>>  Watch
>> Us
>> 
>>
>> 
>>  Read
>> Us
>> 
>>
>


-- 
[image: Logo]

Shmuel Blitz
*Big Data Developer*
www.similarweb.com


Like
Us


Follow
Us


Watch
Us


Read
Us