Hi JianFeng :
<1> All the tez apis is based on Java, could I get the C# version or how can I
implement the C# version based on the tez serice ??
<2>
>>>>>>>>> why does not contains running tasks? How can get the number of
>>>>>>>>> running task and not started tasks?
If you get the data from ATS, then it is not possible to get the running
tasks number. Because these data are pushed to ATS when the vertex is completed
which means there's no running tasks at that time.
But you can use the Tez API to get the status of running vertex where
you can get the number of running tasks.
>>>>>>>>
I am using /ws/v1/timeline/TEZ_DAG_ID/ this ATS API to get DAG execution plan
and also I can get the DAG status , could this status can be running ?? or it
is like vertex info , it is only pushed into ATS when the graph execution has
been finished. If it is so, we also cannot get the DAG info dynamically when it
was running ?
From: Jianfeng (Jeff) Zhang [mailto:[email protected]]
Sent: Saturday, April 4, 2015 8:49 AM
To: Xiaoyong Zhu; [email protected]
Cc: Yifung Lin; HDInsight VS Tooling V-team
Subject: Re: Tez Vertex Info analysis
Hi Xiaoyong,
Here's the javadoc link for tez api.
http://tez.apache.org/releases/0.6.0/tez-api-javadocs/index.html
I would suggest you to check the wordcount example to get started on the tez
api.
https://github.com/apache/tez/blob/master/tez-examples/src/main/java/org/apache/tez/examples/WordCount.java
The main flow is as following:
1. Create TezClient
2. Create DAG
3. Use TezClient to submit DAG (which would return DAGClient to you)
4. Query the DAGClient until the DAG is finished ( here you can get the
vertex status )
Best Regard,
Jeff Zhang
From: Xiaoyong Zhu <[email protected]<mailto:[email protected]>>
Date: Friday, April 3, 2015 at 9:15 PM
To: Jianfeng Zhang <[email protected]<mailto:[email protected]>>,
"[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Cc: Yifung Lin <[email protected]<mailto:[email protected]>>, HDInsight
VS Tooling V-team <[email protected]<mailto:[email protected]>>
Subject: RE: Tez Vertex Info analysis
Yes, we mean the ATS APIs.
What Tez API did you refer to? Are there additional REST APIs to get the Tez
info directly?
Xiaoyong
From: Jianfeng (Jeff) Zhang [mailto:[email protected]]
Sent: Friday, April 3, 2015 5:29 PM
To: [email protected]<mailto:[email protected]>
Cc: Xiaoyong Zhu; Yifung Lin; HDInsight VS Tooling V-team
Subject: Re: Tez Vertex Info analysis
Hi Joe,
What do you mean tez REST API ? Do you mean you get these info through the ATS
(application timeline service ) ?
<1> what does " numTasks" contains ? what are relationships among numTasks,
numberCompletedTasks, numKilledTasks , numFailedTasks, numSucceededTasks ?
numTasks means the number of tasks of this vertex should run. If you
have some knowledge of map reduce, you can think of this as the number of tasks
in mapper/reducer.
numberCompletedTasks means the number of task that has finished.
There're 3 possible states for task finishing: Succeeded/Failed/Killed. So
that means numberCompletedTasks should be equals to the sum of numKilledTasks ,
numFailedTasks, numSucceededTasks
<2> why does not contains running tasks? How can get the number of running task
and not started tasks?
If you get the data from ATS, then it is not possible to get the running
tasks number. Because these data are pushed to ATS when the vertex is completed
which means there's no running tasks at that time.
But you can use the Tez API to get the status of running vertex where
you can get the number of running tasks.
<3> whether any task in this vertex failed will called the whole vertex failed??
Yes, vertex only succeeded when no task is failed/killed. That means
numberCompletedTasks should be equal to numSucceededTasks if the vertex is
succeeded.
Best Regard,
Jeff Zhang
From: "Joe Zhang (SDE)" <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Friday, April 3, 2015 at 3:30 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Cc: Xiaoyong Zhu <[email protected]<mailto:[email protected]>>,
Yifung Lin <[email protected]<mailto:[email protected]>>, HDInsight VS
Tooling V-team <[email protected]<mailto:[email protected]>>
Subject: Tez Vertex Info analysis
Hi Tez experts:
I am using tez REST API to analysis vertex running information, below is
what I get . but I am wandering some concepts
<1> what does " numTasks" contains ? what are relationships among numTasks,
numberCompletedTasks, numKilledTasks , numFailedTasks, numSucceededTasks ?
<2> why does not contains running tasks? How can get the number of running task
and not started tasks?
<3> whether any task in this vertex failed will called the whole vertex failed??
[cid:[email protected]]
Best wishes
Joe zhang
