[
https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227308#comment-16227308
]
Yufei Gu commented on MAPREDUCE-6994:
-------------------------------------
Thanks [[email protected]] for working on this! Looks good
generally. Here are my comments:
# The current design requires users to understand what directory should be
collected by providing multiple directories for "input". For example, users
needs to input
$HADOOP_HOME/share/hadoop/client/*:$HADOOP_HOME/share/hadoop/common/*, etc. The
benefit of this solution is that it is flexible no matter how input directories
organized. However, the directory hierarchy in $HADOOP_HOME is fixed especially
in upstream output, how about providing an option to just input a $HADOOP_HOME
and tool can figure out which sub-directories to get jars? Seems like to make
method {{collectPackages}} recursively traverse the input directory would be
enough since there is predefined whitelist.
# Class {{UploaderException}} should be private. The projects other than Hadoop
shouldn't use this class.
# Three functions are not used in class {{UploaderException}}. They are not
necessary to be there. We can always add them back when we need them.
# {{DefaultJars}} should be a final class with private constructor. Other
options would be make it an enum or an interface.
# I think we need to log an error message(e.g. "too many parameters") under
this condition.
{code} if (parser.getRemainingArgs().length > 0) { printHelp(); return false; }
{code}
# The help message needs some formatting, e.g. "input" needs to be in line with
"fs". "where" needs some work as well. Let's put a real port# in the example,
hdfs://foo.com:8020. The "classpath" should be "input".
{quote}
Usage: mapred frameworkuploader -classpath <input> -fs <filesystem> -target
<framework> [-whitelist <whitelist>] [-blacklist <blacklist>]
where:
input input class path
fs target file system to upload to. Example:
hdfs://foo.com:port
target target file to upload to with a reference name.
Example: /usr/mr-framework.tar.gz#mr-framework
whitelist Regex specifying the full path of jars to include
in the framework tarball. Default is a hardcoded set of jars considered
necessary to include
blacklist Regex specifying the full path of jars to exclude
in the framework tarball. Default is a hardcoded set of jars considered
unnecessary to include
{quote}
# Method {{validateTargetPath}} could throw the UploaderException instead of
return a boolean.
# Logically method {{buildPackage()}} should be part of method {{upload()}}.
How about call {{buildPackage()}} inside of {{upload()}}? In that case,
{{targetStream}} could be local variable instead of a class member.
# Method {{parseWhiteList}} handles both white list and blacklist. You may
split the method to two methods, or even better, one method takes {{whitelist}}
or {{blacklist}} as a parameter and return {{whitelistedFiles}} or
{{blacklistedFiles}} respectively.
# Need a VisibleForTesting annotation for method {{expandEnvironmentVariables}}.
# "maching" -> "matching" in line 269 of class {{TestFrameworkUploader}}
# assertEquals lines in method {{testEnvironmentReplacement}} and
{{testRecursiveEnvironmentReplacement}} aren't necessary to take two lines.
# Method {{expandEnvironmentVariables()}} seems better to be in class {{Shell}}.
> Uploader tool for Distributed Cache Deploy code changes
> -------------------------------------------------------
>
> Key: MAPREDUCE-6994
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6994
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Reporter: Miklos Szegedi
> Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-6994.000.patch
>
>
> The proposal is to create a tool that collects all available jars in the
> Hadoop classpath and adds them to a single tarball file. It then uploads the
> resulting archive to an HDFS directory. This saves the cluster administrator
> from having to set this up manually for Distributed Cache Deploy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]