[
https://issues.apache.org/jira/browse/HADOOP-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798577#action_12798577
]
Todd Lipcon commented on HADOOP-6408:
-------------------------------------
Hi Hemanth,
Yes, I understood that was the original intention, but I don't see how the
memory usage will be very large when compared with the rest of the
JobInProgress object:
In particular, this map consists mostly of flyweight references. The keys of
the map are configuration keys, which are references to String objects already
stored by Configuration. The values are Strings which are created once per
resource that's loaded. So, the hashmap's memory footprint doesn't double the
footprint of the Configuration object by any means - it just has the footprint
of the references themselves.
A moderately full JobConf probably has a couple hundred configuration
parameters, and a fairly full JT has a couple hundred jobs. If each JobConf
takes an additional 32 bytes per key (16 bytes for the references, and 16 bytes
worth of hashmap overhead) then we should be talking a few KB per JobConf, and
5-10MB overall on the JobTracker. To me that seems like a pretty small cost, as
any JT that's managing hundreds of jobs probably has many GB of RAM.
If necessary, I'm happy to write a quick test to measure the memory usage of
Configuration with and without the change.
> Add a /conf servlet to dump running configuration
> -------------------------------------------------
>
> Key: HADOOP-6408
> URL: https://issues.apache.org/jira/browse/HADOOP-6408
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.0
>
> Attachments: hadoop-6408.txt, hadoop-6408.txt, hadoop-6408.txt,
> hadoop-6408.txt, hadoop-6408.txt, hadoop-6408.txt, hadoop-6408.txt,
> hadoop-6408.txt
>
>
> HADOOP-6184 added a command line flag to dump the running configuration. It
> would be great for cluster troubleshooting to provide access to this as a
> servlet, preferably in both JSON and XML formats. But really, any format
> would be better than nothing. This should/could go into all of the daemons.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
