[
https://issues.apache.org/jira/browse/TEZ-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043888#comment-14043888
]
Hitesh Shah edited comment on TEZ-1127 at 6/25/14 6:35 PM:
-----------------------------------------------------------
Comments:
bq. if (amOpts != null && amOpts.length() > 0)
- use isEmpty instead of length ( other places also use length > 0 where
isEmpty() would be better )
Some imports may need cleaning up in TezClientUtils.java given that code has
been moved around.
{code}
- public static final String TEZ_AM_JAVA_OPTS_DEFAULT = " -Xmx1024m ";
+ /** Command line options for the Tez AppMaster process. */
+ public static final String TEZ_AM_LAUNCH_CMD_OPTS = TEZ_AM_PREFIX
+ + "launch.cmd-opts";
+ public static final String TEZ_AM_LAUNCH_CMD_OPTS_DEFAULT =
+ "-Djava.net.preferIPv4Stack=true " +
+ "-Dhadoop.metrics.log.level=WARN " +
+ "-Xmx200m"; // Remove after TEZ-699
{code}
- this is a major backward incompatible change. Is there a reason to change
the property name? Also, there was a default AM_JAVA_OPTS that was set to
-Xmx1024m which has been removed/ignored. In most cases, users will pick AM
java opts from tez-site unless another layer is doing something programmatic
such as MR to Tez conversion. The above forces everyone to change their configs.
{code}
+ public static final String TEZ_AM_LAUNCH_ENV_DEFAULT =
+ "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native";
{code}
- should contain PWD. Incompatible with Windows I believe. Should generate a
platform specific default.
{code}
- tezConf.set(TezConfiguration.TEZ_AM_JAVA_OPTS,
+ tezConf.set(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS,
MRHelpers.getMRAMJavaOpts(tezConf));
{code}
- if the objective is to make users use the tez config properties, this
example is not helping. Maybe just remove this code and let the framework use
whatever was setup in tez-site, etc instead of doing an MR to Tez translation?
was (Author: hitesh):
Comments:
bq. if (amOpts != null && amOpts.length() > 0)
- use isEmpty instead of length ( other places also use length > 0 where
isEmpty() would be better )
Some imports may need cleaning up in TezClientUtils.java given that code has
been moved around.
{code}
- public static final String TEZ_AM_JAVA_OPTS_DEFAULT = " -Xmx1024m ";
+ /** Command line options for the Tez AppMaster process. */
+ public static final String TEZ_AM_LAUNCH_CMD_OPTS = TEZ_AM_PREFIX
+ + "launch.cmd-opts";
+ public static final String TEZ_AM_LAUNCH_CMD_OPTS_DEFAULT =
+ "-Djava.net.preferIPv4Stack=true " +
+ "-Dhadoop.metrics.log.level=WARN " +
+ "-Xmx200m"; // Remove after TEZ-699
{code}
- this is a major backward incompatible change. Is there a reason to change
the property name? Also, there was a default AM_JAVA_OPTS that was set to
-Xmx1024m which has been removed/ignored. In most cases, users will pick AM
java opts from tez-site unless another layer is doing something programmatic
such as MR to Tez conversion. The above forces everyone to change their configs.
{code}
+ public static final String TEZ_AM_LAUNCH_ENV_DEFAULT =
+ "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native";
{code}
- should contain PWD. Incompatible with Windows I believe. Should you
generate a platform specific default.
{code}
- tezConf.set(TezConfiguration.TEZ_AM_JAVA_OPTS,
+ tezConf.set(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS,
MRHelpers.getMRAMJavaOpts(tezConf));
{code}
- if the objective is to make users use the tez config properties, this
example is not helping. Maybe just remove this code and let the framework use
whatever was setup in tez-site, etc instead of doing an MR to Tez translation?
> Add TEZ_TASK_JAVA_OPTS and TEZ_ENV configs to specify values from config
> ------------------------------------------------------------------------
>
> Key: TEZ-1127
> URL: https://issues.apache.org/jira/browse/TEZ-1127
> Project: Apache Tez
> Issue Type: Sub-task
> Affects Versions: 0.4.0
> Reporter: Pala M Muthaia
> Assignee: Bikas Saha
> Attachments: TEZ-1127.1.patch
>
>
> There is no global setting available to specify JVM parameters for tez tasks
> (e.g: specifying native library location with java.library.path). Yarn allows
> specifying application.classpath, but not java opts. The only option right
> now is to set mapreduce.map.java.opts in the tez job config (or
> hive.tez.java.opts for Hive on Tez).
> Suggestion from Bikas, while discussing this in tez-user group, is tez should
> have "TEZ_TASK_JAVA_OPTS so that you don’t have to specify this via mapreduce
> configuration parameters"
--
This message was sent by Atlassian JIRA
(v6.2#6252)
