[
https://issues.apache.org/jira/browse/TEZ-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096093#comment-14096093
]
Bikas Saha commented on TEZ-1347:
---------------------------------
TezAPIHelpers Why does it have to have API is the name. Isnt TezHelpers enough?
Or TezUtils. (Merge other *Utils into TezInternalUtils).
Remove the byte reference from java doc.? User payload should be enough.
Convert a Configuration to compressed user pay load (i.e. byte[]) using
Convert compressed pay load in byte[] to a Conf
Remove the limited private for hiveand pig if this is needed by anyone who
essentially needs to offload MR based pipelines to Tez? Same for other such
cases. @Unstable is fine to keep.
@LimitedPrivate("Hive, Pig")
@Unstable
- public static void translateVertexConfToTez(Configuration conf) {
+ public static void translateMRConfToTez(Configuration conf) {
convertVertexConfToTez(conf);
Wrong javadoc
+ * This is only meant to be used if frameworks are not setting up their own
java options,
+ * and would like to fallback to using java options which may already be
configured for
+ * Hadoop MapReduce mappers. <==================== HERE
*
* Uses mapreduce.admin.reduce.child.java.opts, mapreduce.reduce.java.opts
* and mapreduce.reduce.log.level from config to generate the opts.
@@ -213,7 +305,7 @@ public class MRHelpers {
* @return JAVA_OPTS string to be used in launching the JVM
*/
@SuppressWarnings("deprecation")
- public static String getReduceJavaOpts(Configuration conf) {
+ public static String getJavaOptsForMRReducer(Configuration conf) {
Will numReducers ever be true for anything other than YARNRunner? If that is
the case, then we may not need this code at all for everybody else. Just move
it to YARNRunner?
+ if (numReduces != 0) {
+ conf.setBooleanIfUnset("mapred.reducer.new-api",
......
+ if (numReduces != 0) {
+ ensureNotSet(conf, "mapred.partitioner.class", mode);
Some more javadoc on when to use would help for
MRHelpers.translateMRConfToTez()
> Consolidate MRHelpers
> ---------------------
>
> Key: TEZ-1347
> URL: https://issues.apache.org/jira/browse/TEZ-1347
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: TEZ-1347-initial-review.txt, TEZ-1347.1.txt
>
>
> - Remove methods which don't belong in MRHelpers and potentially move them to
> TezHelpers.
> - Get rid of methods which we don't expect/want users to use.
> - Get rid of multiple variants of the same method, if these exist.
> - Investigate other cleanup in MRHelpers.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
