I see. So you are suggesting the jobmanager to support both /foo/bar and /jobs/:jobid/foo/bar, while the history server only supports the latter.
I was initially thinking having two APIs in jobmanager serving the exact same purpose is a bit tricky. Now I think it's a good point that these two APIs, despite now returning the same results, can return different things in future. Junhan & Yangze, WDYT? Best, Xintong On Fri, Jun 24, 2022 at 3:10 PM Chesnay Schepler <ches...@apache.org> wrote: > This is pretty simple to explain. > > "I want to know the environment the job ran in." -> > /jobs/:jobid/environment > "I want to know the environment the JM ran in." -> /jobmanager/environment > > It's less about the JobID being a parameter, and more of a way for them > to better model the resource they are interested in. > > In the future we could consider the job environment endpoint to return > not just the JM environment, but also those from the CLI/TMs. > > On 24/06/2022 06:37, Xintong Song wrote: > > Whether the job ID is actually used in the end isn't visible after all. > > > > I'm not sure about this. E.g., for an empty session cluster, users have > to > > understand they don't need to provide an actual jobid for requesting > > jobmanager information via rest. > > > > I believe both ways work. I think this is a trade off between a) > explaining > > to history server rest api users how the urls are different from > jobmanager > > and b) explaining to jobmanager rest api users why we need an unused > jobid > > for some of the cases. I'm leaning toward the current approach, because > I'd > > expect a smaller set of history server rest api users than (or even a > > subset of) that of jobmanager. > > > > The plan is to document which (and how) the urls are different from > > jobmanager in the history server page [1]. > > > > Compatibility test indeed should be considered. Thanks for pointing it > out. > > Currently the compatibility of history server rest api is guaranteed by > the > > compatibility of jobmanager rest api. I think the only thing we need is > to > > make sure /foo/bar of jobmanager is identical to /jobs/:jobid/foo/bar of > > history server. We can introduce an interface, as a subtype of > JsonArchivist, > > that archives the json with a path that includes the jobid. Then we can > > test against all relevant handlers as implementations of this interface. > > > > WDYT? > > > > Best, > > > > Xintong > > > > > > [1] > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/advanced/historyserver/#available-requests > > > > > > > > On Thu, Jun 23, 2022 at 5:07 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > >> The addition of the /jobs/:jobid/jobmanager/config / environment > >> exclusively to the HS is a bit of a strange workaround. > >> How do you intend to document those? (and test compatibility)? > >> > >> Why not just add a general /jobs/:jobid/environment endpoint that works > >> just like jobmanager/environment. > >> To me that seems like a cleaner solution. > >> It is somewhat mentioned as an alternative in the FLIP, but I don't > >> understand what is supposed to be confusing about it. > >> Whether the job ID is actually used in the end isn't visible after all. > >> > >> /jobmanager/config could be integrated into /jobs/:jobid/config. > >> > >> The same approach could maybe be used for logs; not really sure yet (not > >> a fan of displaying logs in the HS in the first place). > >> > >> On 23/06/2022 06:55, junhan yang wrote: > >>> Hi all, > >>> > >>> Thank you all for your feedbacks. As far as I can see, it looks like > the > >>> discussion on this FLIP has been converged. > >>> > >>> I will start a new vote thread now. > >>> > >>> Best regards, > >>> Junhan > >>> > >>> Yangze Guo <karma...@gmail.com> 于2022年6月17日周五 14:05写道： > >>> > >>>> Thanks for the input, Jiangang. > >>>> > >>>> I think it's a valid demand to distinguish completed jobs with the > same > >>>> name. > >>>> - If they are different jobs, I think users need to give them > >>>> different meaningful names respectively. > >>>> - If they are exactly the same job, IIUC, what you need is to figure > >>>> out the order. ApplicationId in Yarn might help. But in this case, you > >>>> can just sort them with the start time. > >>>> > >>>> Best, > >>>> Yangze Guo > >>>> > >>>> On Fri, Jun 17, 2022 at 12:13 PM Jiangang Liu < > >> liujiangangp...@gmail.com> > >>>> wrote: > >>>>> Thanks for the FLIP. It is helpful to track detail infos for > completed > >>>> jobs. > >>>>> I want to ask another question. In our environment, sometimes it is > >> hard > >>>> to > >>>>> distinguish jobs since the same job names may appear multi times in > the > >>>>> completed jobs. Because a job may run multi times or different jobs > >> have > >>>>> the same job names. I wonder that wether we can enhance the complete > >> jobs > >>>>> display with more information, such as applicationId and application > >> name > >>>>> in yarn. Maybe it is different in k8s to identify a job. > >>>>> > >>>>> Best > >>>>> Jiangang Liu > >>>>> > >>>>> Yangze Guo <karma...@gmail.com> 于2022年6月17日周五 11:40写道： > >>>>> > >>>>>> Thanks for the feedback, Aitozi and Jing. > >>>>>> > >>>>>>> Are each attempts of the TaskManager or JobManager pods (if failure > >>>>>> occurs) > >>>>>> all be shown in the ui? > >>>>>> > >>>>>> The info of the prior execution attempts will be archived, you could > >>>>>> refer to `ArchivedExecutionVertex$priorExecutions`. > >>>>>> > >>>>>>> It seems that most of these metrics are more interesting to batch > >>>> jobs. > >>>>>> Does it make sense to calculate them for pure streaming jobs too? > >>>>>> > >>>>>> All the proposed metrics will be calculated no matter what the job > >>>> type is. > >>>>>>> Why "duration is less interesting" which is mentioned in the FLIP? > >>>>>> As a first step, we mainly focus on the most interesting status > during > >>>>>> the job lifecycle. The duration of final states like FINISHED and > >>>>>> CANCELED is meaningless, while abnormal conditions like CANCELING > will > >>>>>> not be included at the moment. > >>>>>> > >>>>>>> Could you share your thoughts on "accumulated-busy-time"? It should > >>>>>> describe the time while the task is working as expected, i.e. the > >> happy > >>>>>> path. When do we need it for analytics or diagnosis? > >>>>>> > >>>>>> A task could be busy or idle while it is working. Users may adjust > the > >>>>>> parallelism or the partition key according to the ratio between > them. > >>>>>> > >>>>>> Best, > >>>>>> Yangze Guo > >>>>>> > >>>>>> On Fri, Jun 17, 2022 at 5:08 AM Jing Ge <j...@ververica.com> wrote: > >>>>>>> Hi Junhan > >>>>>>> > >>>>>>> These are must-to-have information for batch processing. Thanks for > >>>>>>> bringing it up. > >>>>>>> > >>>>>>> I have some comments: > >>>>>>> > >>>>>>> 1. It seems that most of these metrics are more interesting to > batch > >>>>>> jobs. > >>>>>>> Does it make sense to calculate them for pure streaming jobs too? > >>>>>>> 2. Why "duration is less interesting" which is mentioned in the > FLIP? > >>>>>>> 3. Could you share your thoughts on "accumulated-busy-time"? It > >>>> should > >>>>>>> describe the time while the task is working as expected, i.e. the > >>>> happy > >>>>>>> path. When do we need it for analytics or diagnosis? > >>>>>>> > >>>>>>> BTW, you might want to optimize the format of the FLIP. Some text > is > >>>>>>> running out of the right border of the wiki page. > >>>>>>> > >>>>>>> Best regards, > >>>>>>> Jing > >>>>>>> > >>>>>>> On Thu, Jun 16, 2022 at 4:40 PM Aitozi <gjying1...@gmail.com> > wrote: > >>>>>>> > >>>>>>>> Thanks Junhan for driving this. It a great improvement for the > >>>> batch > >>>>>> jobs. > >>>>>>>> I'm looking forward to this feature in our internal use case. +1 > >>>> for > >>>>>> it. > >>>>>>>> One more question: > >>>>>>>> > >>>>>>>> Are each attempts of the TaskManager or JobManager pods (if > failure > >>>>>> occurs) > >>>>>>>> all be shown in the ui ? > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Aitozi. > >>>>>>>> > >>>>>>>> Yang Wang <danrtsey...@gmail.com> 于2022年6月16日周四 19:10写道： > >>>>>>>> > >>>>>>>>> Thanks Xintong for the explanation. > >>>>>>>>> > >>>>>>>>> It makes sense to leave the discussion about job result store in > >>>> a > >>>>>>>>> dedicated thread. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Yang > >>>>>>>>> > >>>>>>>>> Xintong Song <tonysong...@gmail.com> 于2022年6月16日周四 13:40写道： > >>>>>>>>> > >>>>>>>>>> My impression of JobResultStore is more about fault tolerance > >>>> and > >>>>>> high > >>>>>>>>>> availability. Using it for providing information to users > >>>> sounds > >>>>>> worth > >>>>>>>>>> exploring. We probably need more time to think it through. > >>>>>>>>>> > >>>>>>>>>> Given that it doesn't conflict with what we have proposed in > >>>> this > >>>>>> FLIP, > >>>>>>>>> I'd > >>>>>>>>>> suggest considering it as a separate thread and exclude it > >>>> from the > >>>>>>>> scope > >>>>>>>>>> of this one. > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> > >>>>>>>>>> Xintong > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Thu, Jun 16, 2022 at 11:43 AM Yang Wang < > >>>> danrtsey...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>>> This is a very useful feature both for finished streaming and > >>>>>> batch > >>>>>>>>> jobs. > >>>>>>>>>>> Except for the WebUI & REST API improvements, I am curious > >>>>>> whether we > >>>>>>>>>> could > >>>>>>>>>>> also integrate some critical information(e.g. latest > >>>> checkpoint) > >>>>>> into > >>>>>>>>> the > >>>>>>>>>>> job result store[1]. > >>>>>>>>>>> I am just feeling this is also somehow related with > >>>> "Completed > >>>>>> Jobs > >>>>>>>>>>> Information Enhancement". > >>>>>>>>>>> And I think the history server is not necessary for all the > >>>>>> scenarios > >>>>>>>>>>> especially when users only want to check the job execution > >>>>>> result. > >>>>>>>>>>> [1]. > >>>>>>>>>>> > >>>>>>>>>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-194%3A+Introduce+the+JobResultStore > >>>>>>>>>>> Best, > >>>>>>>>>>> Yang > >>>>>>>>>>> > >>>>>>>>>>> Xintong Song <tonysong...@gmail.com> 于2022年6月15日周三 15:37写道： > >>>>>>>>>>> > >>>>>>>>>>>> Thanks Junhan, > >>>>>>>>>>>> > >>>>>>>>>>>> +1 for the proposed improvements. > >>>>>>>>>>>> > >>>>>>>>>>>> Best, > >>>>>>>>>>>> > >>>>>>>>>>>> Xintong > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Jun 15, 2022 at 3:16 PM Yangze Guo < > >>>> karma...@gmail.com > >>>>>>>>> wrote: > >>>>>>>>>>>>> Thanks for driving this, Junhan. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I think it's a valuable usability improvement for both > >>>>>> streaming > >>>>>>>>> and > >>>>>>>>>>>>> batch users. Looking forward to the community feedback. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Best, > >>>>>>>>>>>>> Yangze Guo > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Wed, Jun 15, 2022 at 3:10 PM junhan yang < > >>>>>>>>>> yangjunhan1...@gmail.com> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I would like to open a discussion on FLIP-241: > >>>> Completed > >>>>>> Jobs > >>>>>>>>>>>> Information > >>>>>>>>>>>>>> Enhancement. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> As far as we can tell, streaming and batch users have > >>>>>> different > >>>>>>>>>>>> interests > >>>>>>>>>>>>>> in probing a job. As Flink grows into a unified > >>>> streaming & > >>>>>>>> batch > >>>>>>>>>>>>> processor > >>>>>>>>>>>>>> and is adopted by more and more batch users, the user > >>>>>>>> experience > >>>>>>>>> of > >>>>>>>>>>>>>> completed job's inspection has become more and more > >>>>>> important. > >>>>>>>>>> After > >>>>>>>>>>>>> doing > >>>>>>>>>>>>>> several market research, there are several potential > >>>>>>>> improvements > >>>>>>>>>>>>> spotted. > >>>>>>>>>>>>>> The main purpose here is due to the involvement of > >>>> WebUI & > >>>>>> REST > >>>>>>>>> API > >>>>>>>>>>>>>> changes, which should be openly discussed and voted on > >>>> as > >>>>>>>> FLIPs. > >>>>>>>>>>>>>> You can find more details in FLIP-241 document[1]. > >>>> Looking > >>>>>>>>> forward > >>>>>>>>>> to > >>>>>>>>>>>>>> your feedback. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [1] https://cwiki.apache.org/confluence/x/dRD1D > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>> Junhan > >> > >> > >
