Re: Command Executor
Till, You got it, that is exactly what I need. I guess I will have to implement an anonymous module even though the documentation warns you against it :) Oscar On Wednesday, August 9, 2017, 11:41:32 AM EDT, Till Toenshoff <toensh...@me.com> wrote: If I got you right, then what you want is something that has the exact lifetime of the agent process and allows you any kind of a custom API. Maybe you should consider using a module for this. There is this notion of so called anonymous modules. Such module simply coexists with the parent process, in your case the agent.See http://mesos.apache.org/documentation/latest/modules/ for more. Such module in combination with a custom executor might get you where you need to be…. at least that is what I am hoping for. On Aug 9, 2017, at 12:53 PM, Oeg Bizz <oegb...@yahoo.com> wrote: Ben, That is exactly what I did, the problem is that is not being invoked when the mesos-slave starts. I am trying to avoid sending a bogus initial task just to kick it off, but it seems there is no way to tell the mesos-slave to start the executor at startup, is it? Thanks for your reply, Oscar On Tuesday, August 8, 2017, 3:38:17 PM EDT, Benjamin Mahler <bmah...@apache.org> wrote: You're free to write your own long lived executor that can process multiple tasks. The built in executors self-terminate after running the tasks they are launched with. On Tue, Aug 8, 2017 at 2:36 AM, Oeg Bizz <oegb...@yahoo.com> wrote: It is used to notify some services that the agents are out there ready to process data. The service takes into account who and how many are out there and some information from the HB to perform some sort of load balancing. Without the HB the service does not know how many agents are out and tasking is assigned only when an offer is received, but that is not as effective as it should be if the service would know in advance about the cluster's load. Thanks, Oscar On Monday, August 7, 2017, 11:13:10 AM EDT, James Peach <jor...@gmail.com> wrote: > On Aug 5, 2017, at 3:03 AM, Oeg Bizz <oegb...@yahoo.com> wrote: > > I have a framework that relies on information sent by a custom Java Command > Executor; think of some sort of heartbeat. I start getting hearbeats after I > send a task to that mesos-slave, but never before that. That makes me assume > that the CommandExecutor is not started until a task is submitted to be > executed by that agent. Is there a way to tell mesos-slave to start the > ComandExecutor as soon as it starts running? Not AFAIK. Executors are always spawned in order to execute tasks. In your case, what is the heartbeat for, if there are no tasks on the agent? J
Re: Command Executor
Ben, That is exactly what I did, the problem is that is not being invoked when the mesos-slave starts. I am trying to avoid sending a bogus initial task just to kick it off, but it seems there is no way to tell the mesos-slave to start the executor at startup, is it? Thanks for your reply, Oscar On Tuesday, August 8, 2017, 3:38:17 PM EDT, Benjamin Mahler <bmah...@apache.org> wrote: You're free to write your own long lived executor that can process multiple tasks. The built in executors self-terminate after running the tasks they are launched with. On Tue, Aug 8, 2017 at 2:36 AM, Oeg Bizz <oegb...@yahoo.com> wrote: It is used to notify some services that the agents are out there ready to process data. The service takes into account who and how many are out there and some information from the HB to perform some sort of load balancing. Without the HB the service does not know how many agents are out and tasking is assigned only when an offer is received, but that is not as effective as it should be if the service would know in advance about the cluster's load. Thanks, Oscar On Monday, August 7, 2017, 11:13:10 AM EDT, James Peach <jor...@gmail.com> wrote: > On Aug 5, 2017, at 3:03 AM, Oeg Bizz <oegb...@yahoo.com> wrote: > > I have a framework that relies on information sent by a custom Java Command > Executor; think of some sort of heartbeat. I start getting hearbeats after I > send a task to that mesos-slave, but never before that. That makes me assume > that the CommandExecutor is not started until a task is submitted to be > executed by that agent. Is there a way to tell mesos-slave to start the > ComandExecutor as soon as it starts running? Not AFAIK. Executors are always spawned in order to execute tasks. In your case, what is the heartbeat for, if there are no tasks on the agent? J
Re: Command Executor
It is used to notify some services that the agents are out there ready to process data. The service takes into account who and how many are out there and some information from the HB to perform some sort of load balancing. Without the HB the service does not know how many agents are out and tasking is assigned only when an offer is received, but that is not as effective as it should be if the service would know in advance about the cluster's load. Thanks, Oscar On Monday, August 7, 2017, 11:13:10 AM EDT, James Peach <jor...@gmail.com> wrote: > On Aug 5, 2017, at 3:03 AM, Oeg Bizz <oegb...@yahoo.com> wrote: > > I have a framework that relies on information sent by a custom Java Command > Executor; think of some sort of heartbeat. I start getting hearbeats after I > send a task to that mesos-slave, but never before that. That makes me assume > that the CommandExecutor is not started until a task is submitted to be > executed by that agent. Is there a way to tell mesos-slave to start the > ComandExecutor as soon as it starts running? Not AFAIK. Executors are always spawned in order to execute tasks. In your case, what is the heartbeat for, if there are no tasks on the agent? J
Re: Command Executor
I sent this over the weekend so I can see how it would have been missed, does anyone have a suggestion? I have a framework that relies on information sent by a custom Java Command Executor; think of some sort of heartbeat. I start getting hearbeats after I send a task to that mesos-slave, but never before that. That makes me assume that the CommandExecutor is not started until a task is submitted to be executed by that agent. Is there a way to tell mesos-slave to start the ComandExecutor as soon as it starts running? Thanks, Oscar Thanks in advance On Saturday, August 5, 2017, 6:08:37 AM EDT, Oeg Bizz <oegb...@yahoo.com> wrote: I have a framework that relies on information sent by a custom Java Command Executor; think of some sort of heartbeat. I start getting hearbeats after I send a task to that mesos-slave, but never before that. That makes me assume that the CommandExecutor is not started until a task is submitted to be executed by that agent. Is there a way to tell mesos-slave to start the ComandExecutor as soon as it starts running? Thanks, Oscar
Command Executor
I have a framework that relies on information sent by a custom Java Command Executor; think of some sort of heartbeat. I start getting hearbeats after I send a task to that mesos-slave, but never before that. That makes me assume that the CommandExecutor is not started until a task is submitted to be executed by that agent. Is there a way to tell mesos-slave to start the ComandExecutor as soon as it starts running? Thanks, Oscar
Libraries
All, I am continuing developing a Java framework using the Java API rather than the HTTP API (long story here). The ultimate goal is to run the framework on its own container, but in the meantime I am constantly making updates to it. So, I would like to be able to run the framework on my Centos 7 node (no containers) until is ready. I have a mesos-master and mesos-slave running on different containers. How can I run the framework WITHOUT installing mesos on my computer? I want to use the libraries contained in the mesos docker containers, is that possible? I tried mounting a volume and copying the libmesos from the container to that volume and set the MESOS_NATIVE_JAVA_LIBRARY accordingly, but is complaining about not finding the libsvn_delta library. Is there a list somewhere of all the .so files I need to expose? Is what I am trying to do absurd and I should just installed Mesos? I know installing Mesos is the easy answer, but it will be nice to run mesos on containers rather than on the computer. Thanks for your help, Oscar
Re: resourceOffer
Jay, Thanks for the tip about the slack chat, I just joined. Also, I just fixed the problem. I think it was related to me sending two separate accept calls rather than packing both tasks and send just one. I guess once you send an accept(operations, oferIds) you cannot send it again for the same Offer. Oscar On Thursday, March 9, 2017 10:38 AM, Jay Guo <guojiannan1...@gmail.com> wrote: It looks quite weird to me... could you share more details about your scheduler implementation? A code snippet could help a lot. Also, if you want more interactive and prompt communication, please join our slack chat https://mesos-slackin.herokuapp.com/ /J On Thu, Mar 9, 2017 at 5:20 PM, Oeg Bizz <oegb...@yahoo.com> wrote: > Qian, > Added the offer_timeout flag and set it to 2 seconds and the result is > the same. I do not get any offerRescinded() call or anything like that. > > To fix the receiving offers I used the driver.acceptOffers instead of > driver.launchTasks() Looking around the source code I found a TestFramework > source code within Mesos and there is a comment about the launchTask to be > deprecated and the use of acceptOffers was preffered so I changed it to > that. > > Thanks, > > Oscar > > > On Wednesday, March 8, 2017 8:49 PM, Qian Zhang <zhq527...@gmail.com> wrote: > > > It seems the offer has already been removed from Mesos master when you tried > to use it to launch a subsequent task, I think you did not specify > "--offer-timeout" flag when starting Mesos master, right? Did your framework > receive "RESCIND" event from Mesos master for the offer that you want to use > to launch task? > > BTW, how did you resolve the stopping receiving offers issue? > > > Thanks, > Qian Zhang > > On Wed, Mar 8, 2017 at 7:57 PM, Oeg Bizz <oegb...@yahoo.com> wrote: > > Sorry about the continuous rant, but I really would love to get this solved. > I passed the stopping receiving offers, but now the second time I send the > same request I get an error message stating that the offer is no longer > valid even though I sent just one task to the only slave I have running. Ii > am running Mesos 1.1.0 and here are the log files from my last run > > Thanks in advance for all your help. BTW, is there a better way of > submitting questions like a chat, threads, bulletin board? > > Oscar > > > On Wednesday, March 8, 2017 6:31 AM, Oeg Bizz <oegb...@yahoo.com> wrote: > > > Vinod, > I think the previous set is incomplete. I have attached a more compete > set of files. Thanks for the help > > > On Tuesday, March 7, 2017 12:34 PM, Vinod Kone <vinodk...@gmail.com> wrote: > > > Can you share master log? > > @vinodkone > > On Mar 7, 2017, at 2:54 AM, Oeg Bizz <oegb...@yahoo.com> wrote: > > Hi, > I am new at mesos and started exploring its usability for a new project I > will be involved. I wrote an scheduler and an executor and I am able to > send one task which is executed properly. After the first task is finished > I no longer get resourceOffer() invocations to my Scheduler. What am I > missing? If I do not send a task I can the resourceOffer calls consistently > every 5 seconds or so. Also, does Mesos send all of the resources every > time or just a partial list? Thanks in advance for any help, > > Oscar > > > > > > > >
Re: resourceOffer
Qian, Added the offer_timeout flag and set it to 2 seconds and the result is the same. I do not get any offerRescinded() call or anything like that. To fix the receiving offers I used the driver.acceptOffers instead of driver.launchTasks() Looking around the source code I found a TestFramework source code within Mesos and there is a comment about the launchTask to be deprecated and the use of acceptOffers was preffered so I changed it to that. Thanks, Oscar On Wednesday, March 8, 2017 8:49 PM, Qian Zhang <zhq527...@gmail.com> wrote: It seems the offer has already been removed from Mesos master when you tried to use it to launch a subsequent task, I think you did not specify "--offer-timeout" flag when starting Mesos master, right? Did your framework receive "RESCIND" event from Mesos master for the offer that you want to use to launch task? BTW, how did you resolve the stopping receiving offers issue? Thanks,Qian Zhang On Wed, Mar 8, 2017 at 7:57 PM, Oeg Bizz <oegb...@yahoo.com> wrote: Sorry about the continuous rant, but I really would love to get this solved. I passed the stopping receiving offers, but now the second time I send the same request I get an error message stating that the offer is no longer valid even though I sent just one task to the only slave I have running. Ii am running Mesos 1.1.0 and here are the log files from my last run Thanks in advance for all your help. BTW, is there a better way of submitting questions like a chat, threads, bulletin board? Oscar On Wednesday, March 8, 2017 6:31 AM, Oeg Bizz <oegb...@yahoo.com> wrote: Vinod, I think the previous set is incomplete. I have attached a more compete set of files. Thanks for the help On Tuesday, March 7, 2017 12:34 PM, Vinod Kone <vinodk...@gmail.com> wrote: Can you share master log? @vinodkone On Mar 7, 2017, at 2:54 AM, Oeg Bizz <oegb...@yahoo.com> wrote: Hi, I am new at mesos and started exploring its usability for a new project I will be involved. I wrote an scheduler and an executor and I am able to send one task which is executed properly. After the first task is finished I no longer get resourceOffer() invocations to my Scheduler. What am I missing? If I do not send a task I can the resourceOffer calls consistently every 5 seconds or so. Also, does Mesos send all of the resources every time or just a partial list? Thanks in advance for any help, Oscar
Re: resourceOffer
Sorry about the continuous rant, but I really would love to get this solved. I passed the stopping receiving offers, but now the second time I send the same request I get an error message stating that the offer is no longer valid even though I sent just one task to the only slave I have running. Ii am running Mesos 1.1.0 and here are the log files from my last run Thanks in advance for all your help. BTW, is there a better way of submitting questions like a chat, threads, bulletin board? Oscar On Wednesday, March 8, 2017 6:31 AM, Oeg Bizz <oegb...@yahoo.com> wrote: Vinod, I think the previous set is incomplete. I have attached a more compete set of files. Thanks for the help On Tuesday, March 7, 2017 12:34 PM, Vinod Kone <vinodk...@gmail.com> wrote: Can you share master log? @vinodkone On Mar 7, 2017, at 2:54 AM, Oeg Bizz <oegb...@yahoo.com> wrote: Hi, I am new at mesos and started exploring its usability for a new project I will be involved. I wrote an scheduler and an executor and I am able to send one task which is executed properly. After the first task is finished I no longer get resourceOffer() invocations to my Scheduler. What am I missing? If I do not send a task I can the resourceOffer calls consistently every 5 seconds or so. Also, does Mesos send all of the resources every time or just a partial list? Thanks in advance for any help, Oscar logs.tgz Description: application/compressed-tar
Re: resourceOffer
Yes, it sends a TASK_FINISHED when is done and exits. It is using a fractional resource, but will try changing it to a whole number On Tuesday, March 7, 2017 6:03 PM, Qian Zhang <zhq527...@gmail.com> wrote: Is your executor a long running process or it will exit right after the task finishes? And did your executor send "TASK_FINISHED" status update when the task finished? Thanks,Qian Zhang On Wed, Mar 8, 2017 at 3:52 AM, Vinod Kone <vinodk...@apache.org> wrote: Hmm. These logs do not have enough information. All I see is a master starting up and an agent re-registering with a bunch of orphan tasks. I don't see the framework re-registering with the master at all. On Tue, Mar 7, 2017 at 9:41 AM, Oeg Bizz <oegb...@yahoo.com> wrote: Sure, there they are. On Tuesday, March 7, 2017 12:34 PM, Vinod Kone <vinodk...@gmail.com> wrote: Can you share master log? @vinodkone On Mar 7, 2017, at 2:54 AM, Oeg Bizz <oegb...@yahoo.com> wrote: Hi, I am new at mesos and started exploring its usability for a new project I will be involved. I wrote an scheduler and an executor and I am able to send one task which is executed properly. After the first task is finished I no longer get resourceOffer() invocations to my Scheduler. What am I missing? If I do not send a task I can the resourceOffer calls consistently every 5 seconds or so. Also, does Mesos send all of the resources every time or just a partial list? Thanks in advance for any help, Oscar
Re: resourceOffer
On Tuesday, March 7, 2017 12:41 PM, Oeg Bizz <oegb...@yahoo.com> wrote: Sure, there they are. On Tuesday, March 7, 2017 12:34 PM, Vinod Kone <vinodk...@gmail.com> wrote: Can you share master log? @vinodkone On Mar 7, 2017, at 2:54 AM, Oeg Bizz <oegb...@yahoo.com> wrote: Hi, I am new at mesos and started exploring its usability for a new project I will be involved. I wrote an scheduler and an executor and I am able to send one task which is executed properly. After the first task is finished I no longer get resourceOffer() invocations to my Scheduler. What am I missing? If I do not send a task I can the resourceOffer calls consistently every 5 seconds or so. Also, does Mesos send all of the resources every time or just a partial list? Thanks in advance for any help, Oscar
resourceOffer
Hi, I am new at mesos and started exploring its usability for a new project I will be involved. I wrote an scheduler and an executor and I am able to send one task which is executed properly. After the first task is finished I no longer get resourceOffer() invocations to my Scheduler. What am I missing? If I do not send a task I can the resourceOffer calls consistently every 5 seconds or so. Also, does Mesos send all of the resources every time or just a partial list? Thanks in advance for any help, Oscar
