[
https://issues.apache.org/jira/browse/TEZ-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033143#comment-14033143
]
Siddharth Seth commented on TEZ-1080:
-------------------------------------
Usage examples
Complex, configuring multiple properties.
{code}
SortedPartitionedPairConf conf =
SortedPartitionedPairConf.newBuilder("keyClass", "valClass")
.setKeyComparatorClass("comparatorClass")
.enableCompression(null)
.configureOutput("partitionerClassName")
.setSortBufferSize(1024)
.setSorterNumThreads(1)
.build()
.configureInput()
.setCombiner("Combiner", Map<String, String> combinerProperties)
.setShuffleBufferFraction(0.25f)
.build()
.setAdditionalConf("key", "val")
.setAdditionalCons(Map<String, String>)
.build();
conf.getInputPayload();
conf.getOutputPayload();
{code}
Simple, relying primarily on defaults
{code}
SortedPartitionedPairConf conf =
SortedPartitionedPairConf.newBuilder("keyClass", "valClass")
.configureOutput("partitionerClassName").build()
.configureInput().build()
.build();
conf.getInputPayload();
conf.getOutputPayload();
{code}
Optionally, these configuration objects could generated the actual Edge to be
used in the DAGPlan.
Edge conf.createEdge(... ...)
Combiners, Partitioners etc would receive separate key-value pairs, if they
need additional configuration.
The current method of relying on a configuration would no longer work.
[~rohini], [~hagleitn] - any feedback on this API ?
> Configuration for non MR based Inputs/Outputs
> ---------------------------------------------
>
> Key: TEZ-1080
> URL: https://issues.apache.org/jira/browse/TEZ-1080
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-1080.wip.1.txt, TEZ-1080.wip.2.txt
>
>
> De-link configuration from MRHelpers (except for the YARNRunner case), and
> allow for these to be configured easily - exposing necessary setters /
> getters without having to rely on config keys.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
