[jira] [Commented] (TEZ-3385) DAGClient API should be accessible outside of DAG submission
[ https://issues.apache.org/jira/browse/TEZ-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149692#comment-16149692 ] Sergey Shelukhin commented on TEZ-3385: --- Is this only for DAGClient, or for TezClient only? Hive might be interested in the latter for HA/multi-HS2 work (transferring Tez sessions between HS2s). cc [~hagleitn] > DAGClient API should be accessible outside of DAG submission > > > Key: TEZ-3385 > URL: https://issues.apache.org/jira/browse/TEZ-3385 > Project: Apache Tez > Issue Type: New Feature >Reporter: Rohini Palaniswamy > > In PIG-4958, I had to resort to > DAGClient client = new DAGClientImpl(appId, dagID, new > TezConfiguration(conf), null); > This is not good as DAGClientImpl is a internal class and not something users > should be referring to. Tez needs to have an API to give DAGClient given the > appId, dagID and configuration. This is something basic like > JobClient.getJob(String jobID). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (TEZ-3824) MRCombiner creates new JobConf copy per spill
[
https://issues.apache.org/jira/browse/TEZ-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149492#comment-16149492
]
Siddharth Seth edited comment on TEZ-3824 at 8/31/17 7:47 PM:
--
[~jeagles] - I think the change is fine as long as it is within the MRCombiner.
Unfortunately hadoop config does not have a way to retain what has been loaded
in a config object.
Only question I have is whether this is needed only for the new API, or for
both the new apI and the old api. (As is, the patch will send in a null for the
config in case the old API is being used?)
was (Author: sseth):
[~jeagles] - I think the change is fine as long as it is within the MRCombiner.
Unfortunately hadoop config does not have a way to retain what has been loaded
in a config object.
Only question I have is whether this is needed only for the new API, or for
both the new apI and the old api.
> MRCombiner creates new JobConf copy per spill
> -
>
> Key: TEZ-3824
> URL: https://issues.apache.org/jira/browse/TEZ-3824
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3824.001.patch
>
>
> {noformat:title=scope-57(HASH_JOIN) stack trace}
> "SpillThread {scope_60_" #99 daemon prio=5 os_prio=0 tid=0x7f2128d21800
> nid=0x7487 runnable [0x7f21154c4000]
>java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012)
> at
> java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1084)
> at
> java.util.concurrent.ConcurrentHashMap.(ConcurrentHashMap.java:852)
> at org.apache.hadoop.conf.Configuration.(Configuration.java:728)
> - locked <0xd1dc5240> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:442)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:67)
> at
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.(TaskAttemptContextImpl.java:49)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.(TaskInputOutputContextImpl.java:54)
> at
> org.apache.hadoop.mapreduce.task.ReduceContextImpl.(ReduceContextImpl.java:95)
> at
> org.apache.tez.mapreduce.combine.MRCombiner.createReduceContext(MRCombiner.java:237)
> at
> org.apache.tez.mapreduce.combine.MRCombiner.runNewCombiner(MRCombiner.java:181)
> at
> org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:115)
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.runCombineProcessor(ExternalSorter.java:313)
> at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.spill(DefaultSorter.java:937)
> at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.sortAndSpill(DefaultSorter.java:861)
> at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter$SpillThread.run(DefaultSorter.java:780)
> {noformat}
> {code:title=JobConf copy construction for tez}
> public JobContextImpl(Configuration conf, JobID jobId) {
> if (conf instanceof JobConf) {
> this.conf = (JobConf)conf;
> } else {
> --->this.conf = new JobConf(conf);<
> }
> this.jobId = jobId;
> this.credentials = this.conf.getCredentials();
> try {
> this.ugi = UserGroupInformation.getCurrentUser();
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3824) MRCombiner creates new JobConf copy per spill
[
https://issues.apache.org/jira/browse/TEZ-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149492#comment-16149492
]
Siddharth Seth commented on TEZ-3824:
-
[~jeagles] - I think the change is fine as long as it is within the MRCombiner.
Unfortunately hadoop config does not have a way to retain what has been loaded
in a config object.
Only question I have is whether this is needed only for the new API, or for
both the new apI and the old api.
> MRCombiner creates new JobConf copy per spill
> -
>
> Key: TEZ-3824
> URL: https://issues.apache.org/jira/browse/TEZ-3824
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3824.001.patch
>
>
> {noformat:title=scope-57(HASH_JOIN) stack trace}
> "SpillThread {scope_60_" #99 daemon prio=5 os_prio=0 tid=0x7f2128d21800
> nid=0x7487 runnable [0x7f21154c4000]
>java.lang.Thread.State: RUNNABLE
> at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012)
> at
> java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1084)
> at
> java.util.concurrent.ConcurrentHashMap.(ConcurrentHashMap.java:852)
> at org.apache.hadoop.conf.Configuration.(Configuration.java:728)
> - locked <0xd1dc5240> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:442)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:67)
> at
> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl.(TaskAttemptContextImpl.java:49)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.(TaskInputOutputContextImpl.java:54)
> at
> org.apache.hadoop.mapreduce.task.ReduceContextImpl.(ReduceContextImpl.java:95)
> at
> org.apache.tez.mapreduce.combine.MRCombiner.createReduceContext(MRCombiner.java:237)
> at
> org.apache.tez.mapreduce.combine.MRCombiner.runNewCombiner(MRCombiner.java:181)
> at
> org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:115)
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.runCombineProcessor(ExternalSorter.java:313)
> at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.spill(DefaultSorter.java:937)
> at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.sortAndSpill(DefaultSorter.java:861)
> at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter$SpillThread.run(DefaultSorter.java:780)
> {noformat}
> {code:title=JobConf copy construction for tez}
> public JobContextImpl(Configuration conf, JobID jobId) {
> if (conf instanceof JobConf) {
> this.conf = (JobConf)conf;
> } else {
> --->this.conf = new JobConf(conf);<
> }
> this.jobId = jobId;
> this.credentials = this.conf.getCredentials();
> try {
> this.ugi = UserGroupInformation.getCurrentUser();
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3825) Tez UI DAGs page can't query RUNNING or SUBMITTED apps
[ https://issues.apache.org/jira/browse/TEZ-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149118#comment-16149118 ] Jonathan Eagles commented on TEZ-3825: -- [~Sreenath], can you have a look at this patch? It is a small regression from 0.7 > Tez UI DAGs page can't query RUNNING or SUBMITTED apps > -- > > Key: TEZ-3825 > URL: https://issues.apache.org/jira/browse/TEZ-3825 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3825.001.patch > > > status is only a primary filter when a final dag status is set. RUNNING and > SUBMITTED status can't be added as a final status so it must be set to > secondaryFilter -- This message was sent by Atlassian JIRA (v6.4.14#64029)
