Re: Get Task's labels on reconciliation
Hi Altan, In my opinion, the reasons for not including labels in status updates are: (1) Task labels cannot be changed after they are set; since the framework defined the labels in the first place, the framework should know what the labels are. (2) The framework cannot safely "forget" about the labels on a task and rely on getting them back from the master via reconciliation. The reason is that the master might send status updates for tasks the master doesn't have any metadata for. For example: * framework launches task T with labels L on agent A * master fails over, new master elected and recovers from registry * new master waits 10 mins for agent A to reregister * agent A never reregisters, so the framework gets slaveLost() for it * when the framework does an explicit reconciliation for task T on A, it will get TASK_LOST. However, the master cannot fill-in the labels for the task because this master doesn't know anything about the task; task metadata is not stored in the registry, so after failover the new master doesn't have this information. You could make a case for sending labels whenever we can, but for the reasons above, I fear this might encourage questionable coding practices by framework authors. Supporting functionality that only works "most of the time" doesn't seem like a great idea. Neil On Fri, Sep 2, 2016 at 1:34 AM, Altan Alpay wrote: > Hi, > > I got bitten by the same issue. I was setting task labels but unable to > retrieve them back through reconciliation. > If a framework can’t read the labels back, what’s the reason labeling tasks > at first place? > > Is there any other mechanism to read task labels that I am missing? > Preferable through Scheduler API? > > I see that the master exposes labels through /state.json but that’s > terrifiyingly big json blob to deal with. > > Thanks, > Altan > > On 2016-01-29 12:16 (-0700), Benjamin Mahler wrote: >> Ah I'm sorry, not sure if the concept of "latest labels" applies in the> >> same way as "latest health" information for a task, since health is an> >> active property of the task, whereas labels can be specific to updates.> >> >> On Fri, Jan 29, 2016 at 12:09 PM, Benjamin Mahler > >> wrote:> >> >> > I see that the following was filed:> >> > https://issues.apache.org/jira/browse/MESOS-4477> >> >> >> > But this sounds like a bug: if the master knows about the task during> >> > reconciliation, labels should be sent. For example we had this same bug >> > for> >> > health information:> >> > https://issues.apache.org/jira/browse/MESOS-2206> >> >> >> > On Fri, Jan 22, 2016 at 2:10 PM, Andrii Biletskyi <> >> > andrii.bilets...@stealth.ly> wrote:> >> >> >> >> Okay, thanks for your answer. I will rework my algorithm so it doesn't> >> >> rely on task labels/data.> >> >>> >> >> Thanks,> >> >> Andrii> >> >>> >> >> On Fri, Jan 22, 2016 at 11:49 PM, Vinod Kone > >> >> wrote:> >> >>> >> >> >>> On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi <> >> >>> andrii.bilets...@stealth.ly> wrote:> >> >> 8. Scheduler.statusUpdate callback is called for> >> taskId=t1;state=Running. But here I don't receive as part of TaskStatus> >> task labels and taskData, they are just empty.> >> > >> >> >>> Yea, this is expected behavior. Currently master *generates* new status> >> >>> updates (with only a few fields set) when it gets a reconciliation >> >>> request.> >> >> >>> >> >>> >> >> >>
Re: Get Task's labels on reconciliation
Hi, I got bitten by the same issue. I was setting task labels but unable to retrieve them back through reconciliation. If a framework can’t read the labels back, what’s the reason labeling tasks at first place? Is there any other mechanism to read task labels that I am missing? Preferable through Scheduler API? I see that the master exposes labels through /state.json but that’s terrifiyingly big json blob to deal with. Thanks, Altan On 2016-01-29 12:16 (-0700), Benjamin Mahler wrote: > Ah I'm sorry, not sure if the concept of "latest labels" applies in the> > same way as "latest health" information for a task, since health is an> > active property of the task, whereas labels can be specific to updates.> > > On Fri, Jan 29, 2016 at 12:09 PM, Benjamin Mahler > > wrote:> > > > I see that the following was filed:> > > https://issues.apache.org/jira/browse/MESOS-4477> > >> > > But this sounds like a bug: if the master knows about the task during> > > reconciliation, labels should be sent. For example we had this same bug > > for> > > health information:> > > https://issues.apache.org/jira/browse/MESOS-2206> > >> > > On Fri, Jan 22, 2016 at 2:10 PM, Andrii Biletskyi <> > > andrii.bilets...@stealth.ly> wrote:> > >> > >> Okay, thanks for your answer. I will rework my algorithm so it doesn't> > >> rely on task labels/data.> > >>> > >> Thanks,> > >> Andrii> > >>> > >> On Fri, Jan 22, 2016 at 11:49 PM, Vinod Kone > > >> wrote:> > >>> > > >>> On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi <> > >>> andrii.bilets...@stealth.ly> wrote:> > > 8. Scheduler.statusUpdate callback is called for> > taskId=t1;state=Running. But here I don't receive as part of TaskStatus> > task labels and taskData, they are just empty.> > > > > >>> Yea, this is expected behavior. Currently master *generates* new status> > >>> updates (with only a few fields set) when it gets a reconciliation > >>> request.> > > >>> > >>> > >> >
Re: Get Task's labels on reconciliation
Ah I'm sorry, not sure if the concept of "latest labels" applies in the same way as "latest health" information for a task, since health is an active property of the task, whereas labels can be specific to updates. On Fri, Jan 29, 2016 at 12:09 PM, Benjamin Mahler wrote: > I see that the following was filed: > https://issues.apache.org/jira/browse/MESOS-4477 > > But this sounds like a bug: if the master knows about the task during > reconciliation, labels should be sent. For example we had this same bug for > health information: > https://issues.apache.org/jira/browse/MESOS-2206 > > On Fri, Jan 22, 2016 at 2:10 PM, Andrii Biletskyi < > andrii.bilets...@stealth.ly> wrote: > >> Okay, thanks for your answer. I will rework my algorithm so it doesn't >> rely on task labels/data. >> >> Thanks, >> Andrii >> >> On Fri, Jan 22, 2016 at 11:49 PM, Vinod Kone >> wrote: >> >>> >>> On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi < >>> andrii.bilets...@stealth.ly> wrote: >>> 8. Scheduler.statusUpdate callback is called for taskId=t1;state=Running. But here I don't receive as part of TaskStatus task labels and taskData, they are just empty. >>> >>> Yea, this is expected behavior. Currently master *generates* new status >>> updates (with only a few fields set) when it gets a reconciliation request. >>> >> >> >
Re: Get Task's labels on reconciliation
I see that the following was filed: https://issues.apache.org/jira/browse/MESOS-4477 But this sounds like a bug: if the master knows about the task during reconciliation, labels should be sent. For example we had this same bug for health information: https://issues.apache.org/jira/browse/MESOS-2206 On Fri, Jan 22, 2016 at 2:10 PM, Andrii Biletskyi < andrii.bilets...@stealth.ly> wrote: > Okay, thanks for your answer. I will rework my algorithm so it doesn't > rely on task labels/data. > > Thanks, > Andrii > > On Fri, Jan 22, 2016 at 11:49 PM, Vinod Kone wrote: > >> >> On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi < >> andrii.bilets...@stealth.ly> wrote: >> >>> 8. Scheduler.statusUpdate callback is called for >>> taskId=t1;state=Running. But here I don't receive as part of TaskStatus >>> task labels and taskData, they are just empty. >>> >> >> Yea, this is expected behavior. Currently master *generates* new status >> updates (with only a few fields set) when it gets a reconciliation request. >> > >
Re: Get Task's labels on reconciliation
Okay, thanks for your answer. I will rework my algorithm so it doesn't rely on task labels/data. Thanks, Andrii On Fri, Jan 22, 2016 at 11:49 PM, Vinod Kone wrote: > > On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi < > andrii.bilets...@stealth.ly> wrote: > >> 8. Scheduler.statusUpdate callback is called for taskId=t1;state=Running. >> But here I don't receive as part of TaskStatus task labels and taskData, >> they are just empty. >> > > Yea, this is expected behavior. Currently master *generates* new status > updates (with only a few fields set) when it gets a reconciliation request. >
Re: Get Task's labels on reconciliation
On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi < andrii.bilets...@stealth.ly> wrote: > 8. Scheduler.statusUpdate callback is called for taskId=t1;state=Running. > But here I don't receive as part of TaskStatus task labels and taskData, > they are just empty. > Yea, this is expected behavior. Currently master *generates* new status updates (with only a few fields set) when it gets a reconciliation request.
Re: Get Task's labels on reconciliation
Neil, thanks for the link! Regarding task labels - maybe I'm doing something wrong because I can't see task labels/ taskData received upon task reconciliation. Here's what I'm doing: 1. Start scheduler 2. Scheduler asks to launch some task (taskId=t1) 3. Executor successfully launches task, through mesosDriver returns state Running and sets some labels, taskData 4. Scheduler receives status update, persists it somewhere. Important to note that here I *do receive* as part of TaskStatus object valid task labels and taskData 5. After some time I restart the scheduler 6. Scheduler is registered with defined frameworkId 7. Scheduler requests task reconciliation through driver.reconcileTasks(Collections.empty) 8. Scheduler.statusUpdate callback is called for taskId=t1;state=Running. But here I don't receive as part of TaskStatus task labels and taskData, they are just empty. So is it expected behavior? Thanks, Andrii On Fri, Jan 22, 2016 at 9:02 PM, Neil Conway wrote: > Hi Andrii, > > TaskStatus includes the task's labels [1], so what you're trying to do > should work. > > BTW, we recently wrote up some suggestions on how to write highly > available frameworks. The docs will be on the website the next time it > is refreshed; in the mean time, you can find them here: > > https://reviews.apache.org/r/41896/diff/8#1 > > Neil > > [1] > https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1162 > > On Fri, Jan 22, 2016 at 6:49 AM, Andrii Biletskyi > wrote: > > Hi, > > > > I'm using Mesos 0.25. Is there any way to get some more information about > > the task (like label, taskData) > > when requesting task reconciliation? Right now i see that driver returns > > only taskId, slaveId, status, timestamp. > > > > Let me describe my use case so you have a better understanding. > > > > I'm trying to implement HA framework that will manage a cluster of > database > > nodes. > > The scheduler will be run on marathon, and since I'm managing database > nodes > > I want to store mesos state > > there to tolerate scheduler failures. So the workflow is the following: > > User starts Scheduler on marathon providing some generated frameworkId. > > Scheduler registers itself with the given > > frameworkId and immediately starts reconciliation. If it's a fresh start > > nothing has to be done - we are ready to accept > > "add database node" commands. If it's a failover the idea was to inspect > > each task state returned by the reconciliation > > procedure, extract labels to distinguish whether it's a node, responsible > > for storing mesos state, then extract task data > > where we store node's host:port. With that you can initialize your > storage > > and may operate normally. > > > > Thanks in advance, > > Andrii > > >
Re: Get Task's labels on reconciliation
Hi Andrii, TaskStatus includes the task's labels [1], so what you're trying to do should work. BTW, we recently wrote up some suggestions on how to write highly available frameworks. The docs will be on the website the next time it is refreshed; in the mean time, you can find them here: https://reviews.apache.org/r/41896/diff/8#1 Neil [1] https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1162 On Fri, Jan 22, 2016 at 6:49 AM, Andrii Biletskyi wrote: > Hi, > > I'm using Mesos 0.25. Is there any way to get some more information about > the task (like label, taskData) > when requesting task reconciliation? Right now i see that driver returns > only taskId, slaveId, status, timestamp. > > Let me describe my use case so you have a better understanding. > > I'm trying to implement HA framework that will manage a cluster of database > nodes. > The scheduler will be run on marathon, and since I'm managing database nodes > I want to store mesos state > there to tolerate scheduler failures. So the workflow is the following: > User starts Scheduler on marathon providing some generated frameworkId. > Scheduler registers itself with the given > frameworkId and immediately starts reconciliation. If it's a fresh start > nothing has to be done - we are ready to accept > "add database node" commands. If it's a failover the idea was to inspect > each task state returned by the reconciliation > procedure, extract labels to distinguish whether it's a node, responsible > for storing mesos state, then extract task data > where we store node's host:port. With that you can initialize your storage > and may operate normally. > > Thanks in advance, > Andrii >
