Hey Mike,
-----Original Message----- From: <Cayanan>, "Michael D (388J)" <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, April 2, 2013 8:32 AM To: "[email protected]" <[email protected]> Subject: Re: Workflow Questions >Hey Chris, > >I suppose I could. For some reason I didn't think that it was possible to >submit a job to the Resource Manager other than through the Workflow. >In my case, I'm using a listener to pull files out of the RabbitMQ. After >I pull a file out of the queue, I'd like to submit a job to the Resource >Manager to trigger a Workflow Event. How would one submit a job this way? You just use the XmlRpcResourceManagerClient class (if you are writing a Java program to pull the file out of RabbitMQ). If you are pulling out of RabbitMQ via a script, then you can use the JobSubmitter program that ships with Resource Manager. The job config file that you mention: >Under the Resource Manager trunk, in src/main/resources/examples/jobs, I >see 2 xml files that appear to be config files. Do I need to make use of >these to do what I want to do? Is an XML specification of a job that you can pass to JobSubmitter. It really takes these input: * JobInput formatter (Serializer/DeSerializer class for reading and writing Job Input) * JobClass * config That's about it. > >Also, I know that the Resource Manager has it's own queue where it places >jobs when all of its compute nodes are full. In my case (this is for >Costin's pipeline BTW), I don't want jobs being placed in the Resource >Manager queue. I would like for them to remain in the Rabbit Queue. Is the >Resource Manager's "getNodeLoad()" method the best way to check if a node >is full before submitting a job? Yep. Cheers, Chris > >Thanks in advance for your help! > >-Mike > > >On 4/1/13 10:15 PM, "Mattmann, Chris A (388J)" ><[email protected]> wrote: > >>Hey Mike, >> >>I'll bite, can't you use the Resource Manager and its queues/nodes to do >>this? >> >>Cheers, >>Chris >> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>Chris Mattmann, Ph.D. >>Senior Computer Scientist >>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>Office: 171-266B, Mailstop: 171-246 >>Email: [email protected] >>WWW: http://sunset.usc.edu/~mattmann/ >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>Adjunct Assistant Professor, Computer Science Department >>University of Southern California, Los Angeles, CA 90089 USA >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >> >> >>-----Original Message----- >>From: <Cayanan>, "Michael D (388J)" <[email protected]> >>Reply-To: "[email protected]" <[email protected]> >>Date: Monday, April 1, 2013 8:11 PM >>To: "[email protected]" <[email protected]> >>Subject: Workflow Questions >> >>>Hi all, >>> >>> >>>I'm trying to see if anyone knows of a cool, easy to use "Resource >>>Manager" that can control the amount of Workflows running concurrently >>>on >>>a node. Basically, my set up is like this: >>> >>> >>>- I have a Workflow Manager Server configured to run at most 10 Workflow >>>Events concurrently. >>>- I have a bunch of files sitting on a RabbitMQ queue. >>>- I have a listener that is continually monitoring this particular >>>queue. >>>As soon as it sees a file in this queue, it will trigger a Workflow >>>event. >>> >>> >>>Under this scenario, normally if I have 30 files in the RabbitMQ, my >>>listener will trigger 30 Workflow events. However, the Workflow Server >>>would put 20 of these events in its repository queue while the other 10 >>>are running. >>> >>> >>>What I want some "Resource Manager" component to do is just run the 10 >>>Workflow events only and have the other 20 sitting in the RabbitMQ. As >>>soon as 1 event has finished, then trigger another workflow event to >>>keep >>>the number of concurrent workflows running >>> at 10. >>>Eventually, we will want this "Resource Manager" component to be able to >>>point to another node, node 2, where it can decide to trigger another >>>set >>>of 10 Workflow events while node 1 is still running it's 10 Workflow >>>events. >>> >>> >>>Hope this makes sense. Any helpful tips would be much appreciated. >>> >>> >>>Thanks, >>>Mike >> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
