Re: Get Task's labels on reconciliation

[Andrii Biletskyi]

Neil, thanks for the link!

Regarding task labels - maybe I'm doing something wrong because I can't see
task labels/
taskData received upon task reconciliation. Here's what I'm doing:

1. Start scheduler
2. Scheduler asks to launch some task (taskId=t1)
3. Executor successfully launches task, through mesosDriver returns state
Running and sets some labels, taskData
4. Scheduler receives status update, persists it somewhere. Important to
note that here I *do receive* as part of TaskStatus object valid task
labels and taskData
5. After some time I restart the scheduler
6. Scheduler is registered with defined frameworkId
7. Scheduler requests task reconciliation through
driver.reconcileTasks(Collections.empty)
8. Scheduler.statusUpdate callback is called for taskId=t1;state=Running.
But here I don't receive as part of TaskStatus task labels and taskData,
they are just empty.

So is it expected behavior?

Thanks,
Andrii


On Fri, Jan 22, 2016 at 9:02 PM, Neil Conway  wrote:

> Hi Andrii,
>
> TaskStatus includes the task's labels [1], so what you're trying to do
> should work.
>
> BTW, we recently wrote up some suggestions on how to write highly
> available frameworks. The docs will be on the website the next time it
> is refreshed; in the mean time, you can find them here:
>
> https://reviews.apache.org/r/41896/diff/8#1
>
> Neil
>
> [1]
> https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1162
>
> On Fri, Jan 22, 2016 at 6:49 AM, Andrii Biletskyi
>  wrote:
> > Hi,
> >
> > I'm using Mesos 0.25. Is there any way to get some more information about
> > the task (like label, taskData)
> > when requesting task reconciliation? Right now i see that driver returns
> > only taskId, slaveId, status, timestamp.
> >
> > Let me describe my use case so you have a better understanding.
> >
> > I'm trying to implement HA framework that will manage a cluster of
> database
> > nodes.
> > The scheduler will be run on marathon, and since I'm managing database
> nodes
> > I want to store mesos state
> > there to tolerate scheduler failures. So the workflow is the following:
> > User starts Scheduler on marathon providing some generated frameworkId.
> > Scheduler registers itself with the given
> > frameworkId and immediately starts reconciliation. If it's a fresh start
> > nothing has to be done - we are ready to accept
> > "add database node" commands. If it's a failover the idea was to inspect
> > each task state returned by the reconciliation
> > procedure, extract labels to distinguish whether it's a node, responsible
> > for storing mesos state, then extract task data
> > where we store node's host:port. With that you can initialize your
> storage
> > and may operate normally.
> >
> > Thanks in advance,
> > Andrii
> >
>

Re: Get Task's labels on reconciliation

[Neil Conway]

Hi Andrii,

TaskStatus includes the task's labels [1], so what you're trying to do
should work.

BTW, we recently wrote up some suggestions on how to write highly
available frameworks. The docs will be on the website the next time it
is refreshed; in the mean time, you can find them here:

https://reviews.apache.org/r/41896/diff/8#1

Neil

[1] https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1162

On Fri, Jan 22, 2016 at 6:49 AM, Andrii Biletskyi
 wrote:
> Hi,
>
> I'm using Mesos 0.25. Is there any way to get some more information about
> the task (like label, taskData)
> when requesting task reconciliation? Right now i see that driver returns
> only taskId, slaveId, status, timestamp.
>
> Let me describe my use case so you have a better understanding.
>
> I'm trying to implement HA framework that will manage a cluster of database
> nodes.
> The scheduler will be run on marathon, and since I'm managing database nodes
> I want to store mesos state
> there to tolerate scheduler failures. So the workflow is the following:
> User starts Scheduler on marathon providing some generated frameworkId.
> Scheduler registers itself with the given
> frameworkId and immediately starts reconciliation. If it's a fresh start
> nothing has to be done - we are ready to accept
> "add database node" commands. If it's a failover the idea was to inspect
> each task state returned by the reconciliation
> procedure, extract labels to distinguish whether it's a node, responsible
> for storing mesos state, then extract task data
> where we store node's host:port. With that you can initialize your storage
> and may operate normally.
>
> Thanks in advance,
> Andrii
>

Re: Get Task's labels on reconciliation

[Andrii Biletskyi]

Okay, thanks for your answer. I will rework my algorithm so it doesn't rely
on task labels/data.

Thanks,
Andrii

On Fri, Jan 22, 2016 at 11:49 PM, Vinod Kone  wrote:

>
> On Fri, Jan 22, 2016 at 1:07 PM, Andrii Biletskyi <
> andrii.bilets...@stealth.ly> wrote:
>
>> 8. Scheduler.statusUpdate callback is called for taskId=t1;state=Running.
>> But here I don't receive as part of TaskStatus task labels and taskData,
>> they are just empty.
>>
>
> Yea, this is expected behavior. Currently master *generates* new status
> updates (with only a few fields set) when it gets a reconciliation request.
>

Re: Mesos sometimes not allocating the entire cluster

[Klaus Ma]

Can you share the whole log of master? I'll be helpful :).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Thu, Jan 21, 2016 at 11:57 PM, Tom Arnfeld  wrote:

> Guangya - Nope, there's no outstanding offers for any frameworks, the ones
> that are getting offers are responding properly.
>
> Klaus - This was just a sample of logs for a single agent, the cluster has
> at  least ~40 agents at any one time.
>
> On 21 January 2016 at 15:20, Guangya Liu  wrote:
>
>> Can you please help check if some outstanding offers in cluster which
>> does not accept by any framework? You can check this via the endpoint of
>> /master/state.json endpoint.
>>
>> If there are some outstanding offers, you can start the master with a
>> offer_timeout flag to let master rescind some offers if those offers are
>> not accepted by framework.
>>
>> Cited from
>> https://github.com/apache/mesos/blob/master/docs/configuration.md
>>
>> --offer_timeout=VALUE Duration of time before an offer is rescinded from
>> a framework.
>>
>> This helps fairness when running frameworks that hold on to offers, or
>> frameworks that accidentally drop offers.
>>
>> Thanks,
>>
>> Guangya
>>
>> On Thu, Jan 21, 2016 at 9:44 PM, Tom Arnfeld  wrote:
>>
>>> Hi Klaus,
>>>
>>> Sorry I think I explained this badly, these are the logs for one slave
>>> (that's empty) and we can see that it is making offers to some frameworks.
>>> In this instance, the Hadoop framework (and others) are not among those
>>> getting any offers, they get offered nothing. The allocator is deciding to
>>> send offers in a loop to a certain set of frameworks, starving others.
>>>
>>> On 21 January 2016 at 13:17, Klaus Ma  wrote:
>>>
 Yes, it seems Hadoop framework did not consume all offered resources:
 if framework launch task (1 CPUs) on offer (10 CPUs), the other 9 CPUs will
 return back to master (recoverResouces).

 
 Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
 Platform OpenSource Technology, STG, IBM GCG
 +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

 On Thu, Jan 21, 2016 at 6:46 PM, Tom Arnfeld  wrote:

> Thanks everyone!
>
> Stephan - There's a couple of useful points there, will definitely
> give it a read.
>
> Klaus - Thanks, we're running a bunch of different frameworks, in that
> list there's Hadoop MRv1, Apache Spark, Marathon and a couple of home 
> grown
> frameworks we have. In this particular case the Hadoop framework is the
> major concern, as it's designed to continually accept offers until it has
> enough slots it needs. With the example I gave above, we observe that the
> master is never sending any sizeable offers to some of these frameworks
> (the ones with the larger shares), which is where my confusion stems from.
>
> I've attached a snippet of our active master logs which show the
> activity for a single slave (which has no active executors). We can see
> that it's cycling though sending and recovering declined offers from a
> selection of different frameworks (in order) but I can say that not all of
> the frameworks are receiving these offers, in this case that's the Hadoop
> framework.
>
>
> On 21 January 2016 at 00:26, Klaus Ma  wrote:
>
>> Hi Tom,
>>
>> Which framework are you using, e.g. Swarm, Marathon or something
>> else? and which language package are you using?
>>
>> DRF will sort role/framework by allocation ratio, and offer all
>> "available" resources by slave; but if the resources it too small (<
>> 0.1CPU) or the resources was reject/declined by framework, the resources
>> will not offer it until filter timeout. For example, in Swarm 1.0, the
>> default filter timeout 5s (because of go scheduler API); so here is case
>> that may impact the utilisation: the Swarm got one slave with 16 CPUS, 
>> but
>> only launch one container with 1 CPUS; the other 15 CPUS will return back
>>  to master and did not re-offer until filter timeout (5s).
>> I had pull a request to make Swarm's parameters configurable, refer
>> to https://github.com/docker/swarm/pull/1585. I think you can check
>> this case by master log.
>>
>> If any comments, please let me know.
>>
>> 
>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>> On Thu, Jan 21, 2016 at 2:19 AM, Tom Arnfeld  wrote:
>>
>>> Hey,
>>>
>>> I've noticed some interesting behaviour recently when we have lots
>>> of different frameworks connected to our 

Re: Mesos sometimes not allocating the entire cluster

[Tom Arnfeld]

I can’t send the entire log as there’s a lot of activity on the cluster all the 
time, is there anything particular you’re looking for?

> On 22 Jan 2016, at 12:46, Klaus Ma  wrote:
> 
> Can you share the whole log of master? I'll be helpful :).
> 
> 
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer 
> Platform OpenSource Technology, STG, IBM GCG 
> +86-10-8245 4084 | klaus1982...@gmail.com  | 
> http://k82.me 
> On Thu, Jan 21, 2016 at 11:57 PM, Tom Arnfeld  > wrote:
> Guangya - Nope, there's no outstanding offers for any frameworks, the ones 
> that are getting offers are responding properly.
> 
> Klaus - This was just a sample of logs for a single agent, the cluster has at 
>  least ~40 agents at any one time.
> 
> On 21 January 2016 at 15:20, Guangya Liu  > wrote:
> Can you please help check if some outstanding offers in cluster which does 
> not accept by any framework? You can check this via the endpoint of 
> /master/state.json endpoint.
> 
> If there are some outstanding offers, you can start the master with a 
> offer_timeout flag to let master rescind some offers if those offers are not 
> accepted by framework.
> 
> Cited from https://github.com/apache/mesos/blob/master/docs/configuration.md 
> 
> 
> --offer_timeout=VALUE Duration of time before an offer is rescinded from a 
> framework.
> This helps fairness when running frameworks that hold on to offers, or 
> frameworks that accidentally drop offers.
> 
> 
> Thanks,
> 
> Guangya
> 
> On Thu, Jan 21, 2016 at 9:44 PM, Tom Arnfeld  > wrote:
> Hi Klaus,
> 
> Sorry I think I explained this badly, these are the logs for one slave 
> (that's empty) and we can see that it is making offers to some frameworks. In 
> this instance, the Hadoop framework (and others) are not among those getting 
> any offers, they get offered nothing. The allocator is deciding to send 
> offers in a loop to a certain set of frameworks, starving others.
> 
> On 21 January 2016 at 13:17, Klaus Ma  > wrote:
> Yes, it seems Hadoop framework did not consume all offered resources: if 
> framework launch task (1 CPUs) on offer (10 CPUs), the other 9 CPUs will 
> return back to master (recoverResouces).
> 
> 
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer 
> Platform OpenSource Technology, STG, IBM GCG 
> +86-10-8245 4084  | klaus1982...@gmail.com 
>  | http://k82.me 
> On Thu, Jan 21, 2016 at 6:46 PM, Tom Arnfeld  > wrote:
> Thanks everyone!
> 
> Stephan - There's a couple of useful points there, will definitely give it a 
> read.
> 
> Klaus - Thanks, we're running a bunch of different frameworks, in that list 
> there's Hadoop MRv1, Apache Spark, Marathon and a couple of home grown 
> frameworks we have. In this particular case the Hadoop framework is the major 
> concern, as it's designed to continually accept offers until it has enough 
> slots it needs. With the example I gave above, we observe that the master is 
> never sending any sizeable offers to some of these frameworks (the ones with 
> the larger shares), which is where my confusion stems from.
> 
> I've attached a snippet of our active master logs which show the activity for 
> a single slave (which has no active executors). We can see that it's cycling 
> though sending and recovering declined offers from a selection of different 
> frameworks (in order) but I can say that not all of the frameworks are 
> receiving these offers, in this case that's the Hadoop framework.
> 
> 
> On 21 January 2016 at 00:26, Klaus Ma  > wrote:
> Hi Tom,
> 
> Which framework are you using, e.g. Swarm, Marathon or something else? and 
> which language package are you using?
> 
> DRF will sort role/framework by allocation ratio, and offer all "available" 
> resources by slave; but if the resources it too small (< 0.1CPU) or the 
> resources was reject/declined by framework, the resources will not offer it 
> until filter timeout. For example, in Swarm 1.0, the default filter timeout 
> 5s (because of go scheduler API); so here is case that may impact the 
> utilisation: the Swarm got one slave with 16 CPUS, but only launch one 
> container with 1 CPUS; the other 15 CPUS will return back  to master and did 
> not re-offer until filter timeout (5s).
> I had pull a request to make Swarm's parameters configurable, refer to 
> https://github.com/docker/swarm/pull/1585 
> . I think you can check this case 
> by master log.
> 
> If any comments, please let me know.
> 
> 
> 

RE: Framework Id and upgrading mesos versions

[David Kesler]

Ah,  so the only issue there is the fix version on the ticket is wrong.  For 
some reason I thought 0.26.0 had just been released much more recently, so 
(combined with the fix version on the ticket) I had assumed that a patch from 
November would definitely have been included.

At least that’s one mystery solved, thanks.

From: haosdent [mailto:haosd...@gmail.com]
Sent: Thursday, January 21, 2016 8:31 PM
To: user
Subject: Re: Framework Id and upgrading mesos versions

>but I noticed that the code added to fix Mesos-3834 appears in the master 
>branch in github, but not the 0.26.0 branch.
0.26-rc1 checkout since Nov 13,2015 while this patch submit in Nov 24.2015, so 
don't contains this patch.

On Fri, Jan 22, 2016 at 7:19 AM, David Kesler 
> wrote:
I'm attempting to test upgrading from our current version of mesos (0.22.1) to 
the latest.  Even when going only one minor version at a time, I'm running into 
issues due to the lack of framework id in the framework info.

I've been able to replicate the issue reliably.  I started with with a single 
master and slave, with a fresh install of marathon 0.9.0 and mesos 0.22.1, 
wiping out /tmp/mesos on the slave and /mesos and /marathon in zookeeper.  I 
started up a task.  At this point, I can look at 
`/tmp/mesos/meta/slaves/latest/frameworks//framework.info` and verify that there is no 
framework id present in the file.  I then upgraded the master to mesos 0.23.1, 
restarted it, then the slave to 0.23.1 and restarted it, then marathon to 
0.11.1 (which was built against mesos 0.23) and restarted it.  The slave came 
up and recovered just fine.  However the framework.info 
file never gets updated with the framework id.  If I then proceed to upgrade 
the master to 0.24, restart it, then the slave to 0.24 and restart it, the 
slave fails to come up with the following error:

Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: I0121 
17:54:46.409395  9527 main.cpp:187] Version: 0.24.1
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: I0121 
17:54:46.409406  9527 main.cpp:190] Git tag: 0.24.1
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: I0121 
17:54:46.409418  9527 main.cpp:194] Git SHA: 
44873806c2bb55da37e9adbece938274d8cd7c48
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: I0121 
17:54:46.513608  9527 containerizer.cpp:143] Using isolation: 
posix/cpu,posix/mem,filesystem/posix
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: 2016-01-21 
17:54:46,514:9527(0x7f18d63e1700):ZOO_INFO@log_env@712: Client 
environment:zookeeper.version=zookeeper C client 3.4.5
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: 2016-01-21 
17:54:46,514:9527(0x7f18d63e1700):ZOO_INFO@log_env@716: Client 
environment:host.name=dev-sandbox-mesos-slave1
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: 2016-01-21 
17:54:46,514:9527(0x7f18d63e1700):ZOO_INFO@log_env@723: Client 
environment:os.name=Linux
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: 2016-01-21 
17:54:46,514:9527(0x7f18d63e1700):ZOO_INFO@log_env@724: Client 
environment:os.arch=3.13.0-58-generic
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: 2016-01-21 
17:54:46,514:9527(0x7f18d63e1700):ZOO_INFO@log_env@725: Client 
environment:os.version=#97-Ubuntu SMP Wed Jul 8 02:56:15 UTC 2015
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: I0121 
17:54:46.514710  9527 main.cpp:272] Starting Mesos slave
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: I0121 
17:54:46.516090  9542 slave.cpp:190] Slave started on 
1)@10.100.25.112:5051
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: I0121 
17:54:46.516180  9542 slave.cpp:191] Flags at startup: 
--appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" 
--cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" 
--cgroups_hierar
chy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" 
--container_disk_watch_interval="15secs" --containerizers="docker,mesos" 
--default_role="*" --disk_watch_interval="1mins" --docker="docker" 
--docker_kill_orphans="true" --docker_remove_delay="6hrs" --docker_so
cket="/var/run/docker.sock" --docker_stop_timeout="0ns" 
--enforce_container_disk_quota="false" --executor_registration_timeout="5mins" 
--executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" 
--fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
 --gc_disk_headroom="0.1" --hadoop_home="" --help="false" 
--initialize_driver_logging="true" --ip="10.100.25.112" 
--isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" 
--log_dir="/var/log/mesos" --logbufsecs="0" --logging
Jan 21 17:54:46 dev-sandbox-mesos-slave1 mesos-slave[9527]: _level="INFO"