[
https://issues.apache.org/jira/browse/TEZ-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035710#comment-14035710
]
Rohini Palaniswamy commented on TEZ-1080:
-----------------------------------------
bq. Do we want to keep supporting default values for things like
intermediate-output-compression via tez-site.xml ?
Yes. We would like to have cluster wide configuration through tez-site.xml
for compression, shuffle tuning, etc.
bq. Combiners, Partitioners etc would receive separate key-value pairs, if they
need additional configuration.
How about input format/output format?
API looks clean. But in terms of simplicity and how we are going to use it, it
might actually be more complicated for us. For starters, we will have to do
conversion. Code is going to look like this for us.
{code}
SortedPartitionedPairBuilder b =
SortedPartitionedPairConf.newBuilder("keyClass", "valClass");
SortedPartitionedOutputBuilder output =
b.configureOutput("partitionerClassName");
if (conf.get("tez.io.sort.mb") != null) {
output.setSortBufferSize(conf.get("tez.io.sort.mb"));
}
....
{code}
> Configuration for non MR based Inputs/Outputs
> ---------------------------------------------
>
> Key: TEZ-1080
> URL: https://issues.apache.org/jira/browse/TEZ-1080
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-1080.wip.1.txt, TEZ-1080.wip.2.txt
>
>
> De-link configuration from MRHelpers (except for the YARNRunner case), and
> allow for these to be configured easily - exposing necessary setters /
> getters without having to rely on config keys.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
