Re: [Spacewalk-devel] SSM Asynchronous Actions
Miroslav Suchý wrote: Jason Dobies wrote: As I was working on the performance enhancements for the SSM bulk channel subscription change (https://bugzilla.redhat.com/show_bug.cgi?id=469984) I started to wonder if our approach to the SSM in general could use a little work. Given the fact that there can potentially be a large number of servers selected, I think the whole setup lends itself towards an asynchronous model. Quick question: what is the longest time for SSM action execution. I mean either you personally experienced, or customer experienced or whoever else experienced. Do we speak about seconds, minutes or hours? I've hit the 5 minute HTTP timeout before when deleting 1000 systems with the ssm. Mike -- Mike McCune mmccune AT redhat.com Engineering | Portland, OR RHN Satellite | 650.567.9039x79248 ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
[snip] > You have to be doing a lot of systems, but some actions can take a > minute or more per system on a loaded satellite. There's some pl/sql > that needs optimizing me thinks... I haven't looked into the base channel changing, but you can see some of the numbers before and after my child channel changing optimizations here: https://bugzilla.redhat.com/show_bug.cgi?id=469984 I understand deleting a server is a really painful operation as well. > so engineering a solution that will > allow operations that are taking days is the wrong thing to fix IMO :) I completely agree, but that's not the idea. Inevitably larger amounts of systems take longer, so I think making them async in general makes sense. But the second step is revisiting a lot of these implementations. From what I've seen, the approach seems to be doing the same thing as for single servers, just multiple times. There's a lot of room for caching results across the batch as well as some more creative ways of tuning for multiple systems. -- Jason Dobies (jason.dob...@redhat.com) RHN Satellite & Spacewalk RHCE# 805008743336126 Freenode: jdob @ #spacewalk, #spacewalk-devel ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
Jesus M. Rodriguez wrote: > On Wed, Jan 28, 2009 at 10:09 AM, Justin Sherrill wrote: >> Jesus M. Rodriguez wrote: >>> On Tue, Jan 27, 2009 at 5:30 PM, Justin Sherrill >>> wrote: Jesus Rodriguez wrote: > On Tue, Jan 27, 2009 at 04:12:43PM -0500, Jason Dobies wrote: > [snip] > >> Outside of what we track for a request, the big question is how to >> thread off the work. >> >> One possibility is taskomatic, which I'm not entirely familiar with. I >> do see from the wiki that we have tasks scheduled for once a minute, >> which I think is an acceptable delay before one of these SSM actions >> begins to process. > taskomatic can run tasks that poll in any configuration. i.e. once a > minute, daily, etc. > >> I think we have to be able to allow more than one of these SSM actions >> to take place concurrently, however there is a definite upper limit to >> how many we should allow at once. We'll also need a mechanism to time >> out these actions and potentially a way for a user-initiated cancel. I'm >> not sure if it's possible to exercise this sort of set up through >> taskomatic/quartz. Anyone familiar care to comment? > You can start a job using the MessageQueue. :) It was originally > written to send out emails, but we also used it to calcuate the errata > cache > during login. It's as simple creating an appropriate Event class and > an Action (NOT a Struts Action) class. I would vote against the MessageQueue because if you restart tomcat you've lost all of your scheduling. Whereas if you were using taskomatic, you still would have all the queued actions left in the DB. >>> Does that mean that the MessageQueue can't be enhanced to use >>> a database? Just because it doesn't right now doesn't mean it can't. >>> I was merely suggesting the MessageQueue as a starting point to avoid >>> inventing yet another threading system. It wouldn't be difficult >>> to back the queue with a db table. >>> >> We could do that, although after a restart something would have to kick >> off the event in the message queue again (If i understand it correctly, >> could be wrong here). >> >>> And how often do you restart tomcat? on a dev system all the time sure. >>> But in a production server? and what happens if you lose all of the items >>> in the queue? what's the worse that has to happen? you restart them all? >>> all of that is annoying if the number is large and the process is >>> cumbersome. >>> so let's not dismiss it entirely. >> usually not that often. But with some of our operations taking *days* >> on 1,000s of systems it becomes very possible that a restart may be >> needed. > > *days*? that's just wrong and busted :) so engineering a solution that will > allow operations that are taking days is the wrong thing to fix IMO :) > > jesus > You have to be doing a lot of systems, but some actions can take a minute or more per system on a loaded satellite. There's some pl/sql that needs optimizing me thinks... > ___ > Spacewalk-devel mailing list > Spacewalk-devel@redhat.com > https://www.redhat.com/mailman/listinfo/spacewalk-devel ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
On Wed, Jan 28, 2009 at 10:09 AM, Justin Sherrill wrote: > Jesus M. Rodriguez wrote: >> On Tue, Jan 27, 2009 at 5:30 PM, Justin Sherrill wrote: >>> Jesus Rodriguez wrote: On Tue, Jan 27, 2009 at 04:12:43PM -0500, Jason Dobies wrote: [snip] > Outside of what we track for a request, the big question is how to > thread off the work. > > One possibility is taskomatic, which I'm not entirely familiar with. I > do see from the wiki that we have tasks scheduled for once a minute, > which I think is an acceptable delay before one of these SSM actions > begins to process. taskomatic can run tasks that poll in any configuration. i.e. once a minute, daily, etc. > I think we have to be able to allow more than one of these SSM actions > to take place concurrently, however there is a definite upper limit to > how many we should allow at once. We'll also need a mechanism to time > out these actions and potentially a way for a user-initiated cancel. I'm > not sure if it's possible to exercise this sort of set up through > taskomatic/quartz. Anyone familiar care to comment? You can start a job using the MessageQueue. :) It was originally written to send out emails, but we also used it to calcuate the errata cache during login. It's as simple creating an appropriate Event class and an Action (NOT a Struts Action) class. >>> I would vote against the MessageQueue because if you restart tomcat >>> you've lost all of your scheduling. Whereas if you were using >>> taskomatic, you still would have all the queued actions left in the DB. >> >> Does that mean that the MessageQueue can't be enhanced to use >> a database? Just because it doesn't right now doesn't mean it can't. >> I was merely suggesting the MessageQueue as a starting point to avoid >> inventing yet another threading system. It wouldn't be difficult >> to back the queue with a db table. >> > > We could do that, although after a restart something would have to kick > off the event in the message queue again (If i understand it correctly, > could be wrong here). > >> And how often do you restart tomcat? on a dev system all the time sure. >> But in a production server? and what happens if you lose all of the items >> in the queue? what's the worse that has to happen? you restart them all? >> all of that is annoying if the number is large and the process is cumbersome. >> so let's not dismiss it entirely. > > usually not that often. But with some of our operations taking *days* > on 1,000s of systems it becomes very possible that a restart may be > needed. *days*? that's just wrong and busted :) so engineering a solution that will allow operations that are taking days is the wrong thing to fix IMO :) jesus ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
Jesus M. Rodriguez wrote: > On Tue, Jan 27, 2009 at 5:30 PM, Justin Sherrill wrote: >> Jesus Rodriguez wrote: >>> On Tue, Jan 27, 2009 at 04:12:43PM -0500, Jason Dobies wrote: >>> [snip] >>> Outside of what we track for a request, the big question is how to thread off the work. One possibility is taskomatic, which I'm not entirely familiar with. I do see from the wiki that we have tasks scheduled for once a minute, which I think is an acceptable delay before one of these SSM actions begins to process. >>> taskomatic can run tasks that poll in any configuration. i.e. once a >>> minute, daily, etc. >>> I think we have to be able to allow more than one of these SSM actions to take place concurrently, however there is a definite upper limit to how many we should allow at once. We'll also need a mechanism to time out these actions and potentially a way for a user-initiated cancel. I'm not sure if it's possible to exercise this sort of set up through taskomatic/quartz. Anyone familiar care to comment? >>> You can start a job using the MessageQueue. :) It was originally >>> written to send out emails, but we also used it to calcuate the errata cache >>> during login. It's as simple creating an appropriate Event class and >>> an Action (NOT a Struts Action) class. >> I would vote against the MessageQueue because if you restart tomcat >> you've lost all of your scheduling. Whereas if you were using >> taskomatic, you still would have all the queued actions left in the DB. > > Does that mean that the MessageQueue can't be enhanced to use > a database? Just because it doesn't right now doesn't mean it can't. > I was merely suggesting the MessageQueue as a starting point to avoid > inventing yet another threading system. It wouldn't be difficult > to back the queue with a db table. > We could do that, although after a restart something would have to kick off the event in the message queue again (If i understand it correctly, could be wrong here). > And how often do you restart tomcat? on a dev system all the time sure. > But in a production server? and what happens if you lose all of the items > in the queue? what's the worse that has to happen? you restart them all? > all of that is annoying if the number is large and the process is cumbersome. > so let's not dismiss it entirely. usually not that often. But with some of our operations taking *days* on 1,000s of systems it becomes very possible that a restart may be needed. > > jesus > > ___ > Spacewalk-devel mailing list > Spacewalk-devel@redhat.com > https://www.redhat.com/mailman/listinfo/spacewalk-devel ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
On Wed, Jan 28, 2009 at 5:43 AM, Miroslav Suchý wrote: > Jason Dobies wrote: >> >> As I was working on the performance enhancements for the SSM bulk >> channel subscription change >> (https://bugzilla.redhat.com/show_bug.cgi?id=469984) I started to wonder >> if our approach to the SSM in general could use a little work. Given the >> fact that there can potentially be a large number of servers selected, I >> think the whole setup lends itself towards an asynchronous model. > > Quick question: what is the longest time for SSM action execution. I mean > either you personally experienced, or customer experienced or whoever else > experienced. > Do we speak about seconds, minutes or hours? I would say minutes. Long enough to cause the webui to timeout, which is bad. jesus ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
On Wed, Jan 28, 2009 at 1:01 AM, Partha Aji wrote: > >>> Does that mean that the MessageQueue can't be enhanced to use >>> a database? Just because it doesn't right now doesn't mean it can't. >>> I was merely suggesting the MessageQueue as a starting point to avoid >>> inventing yet another threading system. It wouldn't be difficult >>> to back the queue with a db table. >>> >>> And how often do you restart tomcat? on a dev system all the time sure. >>> But in a production server? and what happens if you lose all of the items >>> in the queue? what's the worse that has to happen? you restart them all? >>> all of that is annoying if the number is large and the process is >>> cumbersome. >>> so let's not dismiss it entirely. >> >> Good points, and I agree we should consider serializing the queue items. >> We already have a table for Tasks that can be used with this: >> >> rhnTaskQueue >> Name Null?Type >> - >> >> ORG_IDNOT NULL NUMBER >> TASK_NAME NOT NULL VARCHAR2(64) >> TASK_DATANUMBER >> PRIORITYNUMBER >> EARLIEST NOT NULL DATE >> >> this table is mapped nicely in Hibernate with the class "Task". > > One trouble we'll have with serializing using rhnTaskQueue table might be > with the 'type' of task data. >> TASK_DATANUMBER > Lets take an example here.. Lets say I want to subscribe 100 config > channels to 100 systems. Now I 'd need task data to be a list of config > channels... For other operations I need task data to be something else may > be strings... One idea is to make task data a blob that stores json ed value > (I don't like this but it may be workable) in the DB... If the schema needs changing, I'm ok with making the necessary changes. > I wonder if using something like MQ series or JMS where issues like > serialization to hard drive is taken care of is a better solution than a > home grown one.. -1 let's not throw in the sledgehammer yet. :) > > In memory message queue is a good in the sense it'll let us pass in what > ever we want as TaskData. But I am not sure I like the temporary nature... > > But of all the solutions here, I think it would be easier for us to just > reuse rhnServerAction and that infrastructure.. Config* stuff is built in > ... We could probably reuse the Event History UI to do the failed/pending > actions/ rescheduling, all that is built in. Its almost like 1 query might > to be changed for backend to ignore ssm-scheduled task form > rhnServerAction.. (I am sure I am trivializing by saying 1 query but it > shouldn;t be too difficult to work around that,. ) Sounds reasonable, definitely worth looking into. ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
I dunno if its the longest but changing the base channel for a System used to take ~30 secs per system.. So if the user asked the base channel to be changed for 100 systems its almost 50 min to get that done easily timing out the request.. Partha Miroslav Suchý wrote: Jason Dobies wrote: As I was working on the performance enhancements for the SSM bulk channel subscription change (https://bugzilla.redhat.com/show_bug.cgi?id=469984) I started to wonder if our approach to the SSM in general could use a little work. Given the fact that there can potentially be a large number of servers selected, I think the whole setup lends itself towards an asynchronous model. Quick question: what is the longest time for SSM action execution. I mean either you personally experienced, or customer experienced or whoever else experienced. Do we speak about seconds, minutes or hours? ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
Jason Dobies wrote: As I was working on the performance enhancements for the SSM bulk channel subscription change (https://bugzilla.redhat.com/show_bug.cgi?id=469984) I started to wonder if our approach to the SSM in general could use a little work. Given the fact that there can potentially be a large number of servers selected, I think the whole setup lends itself towards an asynchronous model. Quick question: what is the longest time for SSM action execution. I mean either you personally experienced, or customer experienced or whoever else experienced. Do we speak about seconds, minutes or hours? -- Miroslav Suchy RHN Satellite Engineering, Red Hat ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
> But of all the solutions here, I think it would be easier for us to just > reuse rhnServerAction and that infrastructure.. Config* stuff is built > in ... We could probably reuse the Event History UI to do the > failed/pending actions/ rescheduling, all that is built in. Its almost > like 1 query might to be changed for backend to ignore ssm-scheduled > task form rhnServerAction.. (I am sure I am trivializing by saying 1 > query but it shouldn;t be too difficult to work around that,. ) First off, thanks for all the feedback. I'm taking chunks from various e-mails and consolidating my replies. One thing I like about utilizing rhnServerAction is that if I remember correctly (admittedly my head's a bit cloudy right now) that is the table used to track things done to clients. If we get the mapping right between "SSM bulk operation" and rhnServerAction, we can use it for server-side things (delete server, change channel subs) as well as server-client things (package install). > The other nice thing about using the MessageQueue is that is running > *within* tomcat so if we wanted to show status in the GUI we can talk > straight to the queue in memory without having to go out over xmlrpc or > to a DB table in a polling mechanism to track progress. I'm not sure I follow what this gets us, but that may be due to assumptions I'm making in my head that I haven't properly communicated. I was imagining a single paginated page that shows all things done through the SSM, meaning we'd see both complete and in progress side by side: Change Channel Subs -- In Progress -- admin -- 1/28 4:17am Delete Servers -- Completed-- admin -- 1/27 11:04pm Delete Servers -- Timed Out-- admin -- 1/27 9:41pm Where clicking on an individual item shows more details about it. In both this idea and my above comment on the rhnServerAction, I'm thinking in terms of solving as many cases as possible with the same code. > Right now taskomatic doesn't allow on demand scheduling. It reads > the tasks defined in the config only. But it wouldn't be hard to add > say a XMLRPC server to Taskomatic (like we did with search server) > to send jobs to Taskomatic or adding the entries to a task table. I was thinking something like this as well. Quartz seems to be able to support all of what we need, it's just a matter of throwing some hooks to it into taskomatic and exposing them. This isn't me saying it's a better approach than the Message Queue; I don't know enough about it (read: didn't really know it existed until this thread) yet to comment on that. Would the MessageQueue support: - Setting a timeout on the operation so it's not spinning around forever. - Canceling a job in progress. - Limiting the number of current jobs (potentially by user and a global number of concurrent SSM jobs). I'm not sure Quartz supports that either, just something I'm trying to keep in mind. My understanding of SSM usage is that I don't think we should expect a dozen concurrent SSM jobs, but still something I'd like to code defensively around. So given the last case, if a new SSM request comes in and we check and see X many in progress ones, with taskomatic we could reschedule it to occur later. I'm guessing in the MessageQueue we could just sleep in the thread that's spawned, but I'm just kinda thinking out loud here. > I would vote against the MessageQueue because if you restart tomcat > you've lost all of your scheduling. Whereas if you were using > taskomatic, you still would have all the queued actions left in the DB. I don't expect these things to sit in the queue long. I'm looking to make the UI async, however it is not where we'd have the "schedule this for some time in the future" logic. So after the user confirms the action, I'm expecting it to start in pretty short time. Given that, I'm not so worried about losing queued items as I am interrupting an existing, long running batch of operations, which we really wouldn't be able to reliably restart on our own anyway. > We'd have to upgrade to a full blown appserver for EJBs wouldn't we? > So for now I'd say no to that. Totally agree, just throwing it out there. :) -- Jason Dobies (jason.dob...@redhat.com) RHN Satellite & Spacewalk RHCE# 805008743336126 Freenode: jdob @ #spacewalk, #spacewalk-devel ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
Does that mean that the MessageQueue can't be enhanced to use a database? Just because it doesn't right now doesn't mean it can't. I was merely suggesting the MessageQueue as a starting point to avoid inventing yet another threading system. It wouldn't be difficult to back the queue with a db table. And how often do you restart tomcat? on a dev system all the time sure. But in a production server? and what happens if you lose all of the items in the queue? what's the worse that has to happen? you restart them all? all of that is annoying if the number is large and the process is cumbersome. so let's not dismiss it entirely. Good points, and I agree we should consider serializing the queue items. We already have a table for Tasks that can be used with this: rhnTaskQueue Name Null?Type - ORG_IDNOT NULL NUMBER TASK_NAME NOT NULL VARCHAR2(64) TASK_DATANUMBER PRIORITYNUMBER EARLIEST NOT NULL DATE this table is mapped nicely in Hibernate with the class "Task". One trouble we'll have with serializing using rhnTaskQueue table might be with the 'type' of task data. > TASK_DATANUMBER Lets take an example here.. Lets say I want to subscribe 100 config channels to 100 systems. Now I 'd need task data to be a list of config channels... For other operations I need task data to be something else may be strings... One idea is to make task data a blob that stores json ed value (I don't like this but it may be workable) in the DB... I wonder if using something like MQ series or JMS where issues like serialization to hard drive is taken care of is a better solution than a home grown one.. In memory message queue is a good in the sense it'll let us pass in what ever we want as TaskData. But I am not sure I like the temporary nature... But of all the solutions here, I think it would be easier for us to just reuse rhnServerAction and that infrastructure.. Config* stuff is built in ... We could probably reuse the Event History UI to do the failed/pending actions/ rescheduling, all that is built in. Its almost like 1 query might to be changed for backend to ignore ssm-scheduled task form rhnServerAction.. (I am sure I am trivializing by saying 1 query but it shouldn;t be too difficult to work around that,. ) Partha ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
Jesus M. Rodriguez wrote: On Tue, Jan 27, 2009 at 5:30 PM, Justin Sherrill wrote: Jesus Rodriguez wrote: On Tue, Jan 27, 2009 at 04:12:43PM -0500, Jason Dobies wrote: [snip] Outside of what we track for a request, the big question is how to thread off the work. One possibility is taskomatic, which I'm not entirely familiar with. I do see from the wiki that we have tasks scheduled for once a minute, which I think is an acceptable delay before one of these SSM actions begins to process. taskomatic can run tasks that poll in any configuration. i.e. once a minute, daily, etc. I think we have to be able to allow more than one of these SSM actions to take place concurrently, however there is a definite upper limit to how many we should allow at once. We'll also need a mechanism to time out these actions and potentially a way for a user-initiated cancel. I'm not sure if it's possible to exercise this sort of set up through taskomatic/quartz. Anyone familiar care to comment? You can start a job using the MessageQueue. :) It was originally written to send out emails, but we also used it to calcuate the errata cache during login. It's as simple creating an appropriate Event class and an Action (NOT a Struts Action) class. I would vote against the MessageQueue because if you restart tomcat you've lost all of your scheduling. Whereas if you were using taskomatic, you still would have all the queued actions left in the DB. Does that mean that the MessageQueue can't be enhanced to use a database? Just because it doesn't right now doesn't mean it can't. I was merely suggesting the MessageQueue as a starting point to avoid inventing yet another threading system. It wouldn't be difficult to back the queue with a db table. And how often do you restart tomcat? on a dev system all the time sure. But in a production server? and what happens if you lose all of the items in the queue? what's the worse that has to happen? you restart them all? all of that is annoying if the number is large and the process is cumbersome. so let's not dismiss it entirely. Good points, and I agree we should consider serializing the queue items. We already have a table for Tasks that can be used with this: rhnTaskQueue Name Null?Type - ORG_IDNOT NULL NUMBER TASK_NAME NOT NULL VARCHAR2(64) TASK_DATA NUMBER PRIORITY NUMBER EARLIEST NOT NULL DATE this table is mapped nicely in Hibernate with the class "Task". Right now this is used by taskomatic to pickup certain rows that have the right "task_name" value but there isn't anything saying we couldn't have the MessageQueue handle other types of tasks with different values in "task_name" and use this table to track if it has work todo. The other nice thing about using the MessageQueue is that is running *within* tomcat so if we wanted to show status in the GUI we can talk straight to the queue in memory without having to go out over xmlrpc or to a DB table in a polling mechanism to track progress. Taskomatic is great for 'scheduled' events that need to happen outside the scope of a user's GUI action, but we did build the MessageQueue specifically for projects just like what we are describing here. Mike -- Mike McCune mmccune AT redhat.com Engineering | Portland, OR RHN Satellite | 650.567.9039x79248 ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
On Tue, Jan 27, 2009 at 5:30 PM, Justin Sherrill wrote: > Jesus Rodriguez wrote: >> On Tue, Jan 27, 2009 at 04:12:43PM -0500, Jason Dobies wrote: >> [snip] >> >>> Outside of what we track for a request, the big question is how to >>> thread off the work. >>> >>> One possibility is taskomatic, which I'm not entirely familiar with. I >>> do see from the wiki that we have tasks scheduled for once a minute, >>> which I think is an acceptable delay before one of these SSM actions >>> begins to process. >> >> taskomatic can run tasks that poll in any configuration. i.e. once a >> minute, daily, etc. >> >>> I think we have to be able to allow more than one of these SSM actions >>> to take place concurrently, however there is a definite upper limit to >>> how many we should allow at once. We'll also need a mechanism to time >>> out these actions and potentially a way for a user-initiated cancel. I'm >>> not sure if it's possible to exercise this sort of set up through >>> taskomatic/quartz. Anyone familiar care to comment? >> >> You can start a job using the MessageQueue. :) It was originally >> written to send out emails, but we also used it to calcuate the errata cache >> during login. It's as simple creating an appropriate Event class and >> an Action (NOT a Struts Action) class. > > I would vote against the MessageQueue because if you restart tomcat > you've lost all of your scheduling. Whereas if you were using > taskomatic, you still would have all the queued actions left in the DB. Does that mean that the MessageQueue can't be enhanced to use a database? Just because it doesn't right now doesn't mean it can't. I was merely suggesting the MessageQueue as a starting point to avoid inventing yet another threading system. It wouldn't be difficult to back the queue with a db table. And how often do you restart tomcat? on a dev system all the time sure. But in a production server? and what happens if you lose all of the items in the queue? what's the worse that has to happen? you restart them all? all of that is annoying if the number is large and the process is cumbersome. so let's not dismiss it entirely. jesus ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
On Tue, Jan 27, 2009 at 4:45 PM, Jason Dobies wrote: > Just a quick followup, from my reading Quartz does seem to support a lot > of these needs, including thread pooling and canceling tasks. I'm > willing to bet they support timeouts too. > > I'm not sure how much of this we expose in Taskomatic, such as the > ability to schedule a task without having it pre-defined in the rhn conf > files, so I'd still like to hear back from our resident taskomatic guru > on initial thoughts of how feasible this would be. Right now taskomatic doesn't allow on demand scheduling. It reads the tasks defined in the config only. But it wouldn't be hard to add say a XMLRPC server to Taskomatic (like we did with search server) to send jobs to Taskomatic or adding the entries to a task table. Partha brings up a good point about trying to reuse the rhnServerActions, something to look into. I think I like the idea of adding an api to taskomatic (for doing things on demand) and combining that with a database. The api allows you to do it NOW versus waiting for taskomatic to poll for jobs. jesus ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
I would vote against the MessageQueue because if you restart tomcat you've lost all of your scheduling. Whereas if you were using taskomatic, you still would have all the queued actions left in the DB. +1 I jsherrill were dicussing how we'd implement this I started with Table Queue -> queue_id|Queue of Servers |Action_Type Table QueuArgs -> queue_id|key|value Then I realize ugh ... Value can be an int, String , list of ids UGH... I then realize we already have rhnServerAction.. infrastructure to schedule the damn thing... We should just leverage that (may be add a flag that says taskomatic) and change the backend to ignore processing those rows that have the taskomatic flag Partha ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
Jesus Rodriguez wrote: > On Tue, Jan 27, 2009 at 04:12:43PM -0500, Jason Dobies wrote: > [snip] > >> Outside of what we track for a request, the big question is how to >> thread off the work. >> >> One possibility is taskomatic, which I'm not entirely familiar with. I >> do see from the wiki that we have tasks scheduled for once a minute, >> which I think is an acceptable delay before one of these SSM actions >> begins to process. > > taskomatic can run tasks that poll in any configuration. i.e. once a > minute, daily, etc. > >> I think we have to be able to allow more than one of these SSM actions >> to take place concurrently, however there is a definite upper limit to >> how many we should allow at once. We'll also need a mechanism to time >> out these actions and potentially a way for a user-initiated cancel. I'm >> not sure if it's possible to exercise this sort of set up through >> taskomatic/quartz. Anyone familiar care to comment? > > You can start a job using the MessageQueue. :) It was originally > written to send out emails, but we also used it to calcuate the errata cache > during login. It's as simple creating an appropriate Event class and > an Action (NOT a Struts Action) class. I would vote against the MessageQueue because if you restart tomcat you've lost all of your scheduling. Whereas if you were using taskomatic, you still would have all the queued actions left in the DB. -Justin > >> Otherwise, outside of EJBs, I'm not sure of any other options (besides >> just spinning off my own threads, but I doubt we want to go that >> route). > > The above mechanism is effectively a new thread :) > ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
On Tue, Jan 27, 2009 at 05:11:19PM -0500, Jesus Rodriguez wrote: > On Tue, Jan 27, 2009 at 04:12:43PM -0500, Jason Dobies wrote: > [snip] > > > Outside of what we track for a request, the big question is how to > > thread off the work. > > > > One possibility is taskomatic, which I'm not entirely familiar with. I > > do see from the wiki that we have tasks scheduled for once a minute, > > which I think is an acceptable delay before one of these SSM actions > > begins to process. > > taskomatic can run tasks that poll in any configuration. i.e. once a > minute, daily, etc. > > > I think we have to be able to allow more than one of these SSM actions > > to take place concurrently, however there is a definite upper limit to > > how many we should allow at once. We'll also need a mechanism to time > > out these actions and potentially a way for a user-initiated cancel. I'm > > not sure if it's possible to exercise this sort of set up through > > taskomatic/quartz. Anyone familiar care to comment? > > You can start a job using the MessageQueue. :) It was originally > written to send out emails, but we also used it to calcuate the errata cache > during login. It's as simple creating an appropriate Event class and > an Action (NOT a Struts Action) class. Look at ./code/src/com/redhat/rhn/frontend/events/ for Event classes. So you can look at UpdateErrataCacheAction and UpdateErrataCacheEvent. Define your own pairs and you can easily spawn your own things in the background. > > Otherwise, outside of EJBs, I'm not sure of any other options (besides > > just spinning off my own threads, but I doubt we want to go that > > route). > > The above mechanism is effectively a new thread :) We'd have to upgrade to a full blown appserver for EJBs wouldn't we? So for now I'd say no to that. -- jesus m. rodriguez| jes...@redhat.com sr. software engineer | irc: zeus rhn satellite & spacewalk | 919.754.4413 (w) rhce # 805008586930012| 919.623.0080 (c) +---+ | "Those who cannot learn from history | | are doomed to repeat it." | | -- George Santayana | +---+ ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
On Tue, Jan 27, 2009 at 04:12:43PM -0500, Jason Dobies wrote: [snip] > Outside of what we track for a request, the big question is how to > thread off the work. > > One possibility is taskomatic, which I'm not entirely familiar with. I > do see from the wiki that we have tasks scheduled for once a minute, > which I think is an acceptable delay before one of these SSM actions > begins to process. taskomatic can run tasks that poll in any configuration. i.e. once a minute, daily, etc. > I think we have to be able to allow more than one of these SSM actions > to take place concurrently, however there is a definite upper limit to > how many we should allow at once. We'll also need a mechanism to time > out these actions and potentially a way for a user-initiated cancel. I'm > not sure if it's possible to exercise this sort of set up through > taskomatic/quartz. Anyone familiar care to comment? You can start a job using the MessageQueue. :) It was originally written to send out emails, but we also used it to calcuate the errata cache during login. It's as simple creating an appropriate Event class and an Action (NOT a Struts Action) class. > Otherwise, outside of EJBs, I'm not sure of any other options (besides > just spinning off my own threads, but I doubt we want to go that > route). The above mechanism is effectively a new thread :) -- jesus m. rodriguez| jes...@redhat.com sr. software engineer | irc: zeus rhn satellite & spacewalk | 919.754.4413 (w) rhce # 805008586930012| 919.623.0080 (c) +---+ | "Those who cannot learn from history | | are doomed to repeat it." | | -- George Santayana | +---+ ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] SSM Asynchronous Actions
Just a quick followup, from my reading Quartz does seem to support a lot of these needs, including thread pooling and canceling tasks. I'm willing to bet they support timeouts too. I'm not sure how much of this we expose in Taskomatic, such as the ability to schedule a task without having it pre-defined in the rhn conf files, so I'd still like to hear back from our resident taskomatic guru on initial thoughts of how feasible this would be. On Tue, 2009-01-27 at 16:12 -0500, Jason Dobies wrote: > As I was working on the performance enhancements for the SSM bulk > channel subscription change > (https://bugzilla.redhat.com/show_bug.cgi?id=469984) I started to wonder > if our approach to the SSM in general could use a little work. Given the > fact that there can potentially be a large number of servers selected, I > think the whole setup lends itself towards an asynchronous model. > > I see it going something like the following: > - At least for the first pass, use the workflow of the existing SSM > pages as they are today (ported to Java, and we can reevaluate each > instance as needed). > > - After selecting to confirm whatever action is being done, instead of > waiting for the action to finish before returning, drop the user to a > generic "SSM Action History" (action may be a bad choice of words here, > but I'm open) page. > > - This page contains a list of each thing the user did against systems > in the SSM. The list can contain: > -- Rough description of what the action was (e.g. "Change channel > subscriptions" or "Delete servers"). Perhaps even editable by the user > at the time of confirming the action. > -- Current status of the action (e.g. "In Progress", "Completed", "Timed > Out", "Cancelled") > -- Timestamp of when started and/or completed and/or last updated. > -- Progress complete. Even if this is just done in increments of, say, > 5%, I'd like to be able to show that the action is at least doing > something. Chunking the updates to this will prevent an update per > server in the set and be able to provide feedback without adding too > much extra load to the DB. > -- Some sort of link to the individual event scheduled on each server, > if applicable. We might even be able to have some logic that looks at > the individual events so when viewing the details of one of these SSM > actions, we can see which servers ran into errors. I think with a little > extra effort this sort of thing can provide much more visibility into > SSM operations than currently provided. This would also be pretty > important given the volatility of the servers in the SSM; since they > aren't named (in other words, reasonably static) groups, I think it'd be > helpful to keep the association of which SSM actions go with which > servers. > > We could even throw a quick status near the SSM "Manage" and "Clear" > buttons to indicate how many actions are currently in progress, so as > the user continues to navigate around they have a quick indicator of if > an action completes. > > Outside of what we track for a request, the big question is how to > thread off the work. > > One possibility is taskomatic, which I'm not entirely familiar with. I > do see from the wiki that we have tasks scheduled for once a minute, > which I think is an acceptable delay before one of these SSM actions > begins to process. > > I think we have to be able to allow more than one of these SSM actions > to take place concurrently, however there is a definite upper limit to > how many we should allow at once. We'll also need a mechanism to time > out these actions and potentially a way for a user-initiated cancel. I'm > not sure if it's possible to exercise this sort of set up through > taskomatic/quartz. Anyone familiar care to comment? > > Otherwise, outside of EJBs, I'm not sure of any other options (besides > just spinning off my own threads, but I doubt we want to go that > route). > > Thoughts? > > - J > > > -- Jason Dobies (jason.dob...@redhat.com) RHN Satellite & Spacewalk RHCE# 805008743336126 Freenode: jdob @ #spacewalk, #spacewalk-devel ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
[Spacewalk-devel] SSM Asynchronous Actions
As I was working on the performance enhancements for the SSM bulk channel subscription change (https://bugzilla.redhat.com/show_bug.cgi?id=469984) I started to wonder if our approach to the SSM in general could use a little work. Given the fact that there can potentially be a large number of servers selected, I think the whole setup lends itself towards an asynchronous model. I see it going something like the following: - At least for the first pass, use the workflow of the existing SSM pages as they are today (ported to Java, and we can reevaluate each instance as needed). - After selecting to confirm whatever action is being done, instead of waiting for the action to finish before returning, drop the user to a generic "SSM Action History" (action may be a bad choice of words here, but I'm open) page. - This page contains a list of each thing the user did against systems in the SSM. The list can contain: -- Rough description of what the action was (e.g. "Change channel subscriptions" or "Delete servers"). Perhaps even editable by the user at the time of confirming the action. -- Current status of the action (e.g. "In Progress", "Completed", "Timed Out", "Cancelled") -- Timestamp of when started and/or completed and/or last updated. -- Progress complete. Even if this is just done in increments of, say, 5%, I'd like to be able to show that the action is at least doing something. Chunking the updates to this will prevent an update per server in the set and be able to provide feedback without adding too much extra load to the DB. -- Some sort of link to the individual event scheduled on each server, if applicable. We might even be able to have some logic that looks at the individual events so when viewing the details of one of these SSM actions, we can see which servers ran into errors. I think with a little extra effort this sort of thing can provide much more visibility into SSM operations than currently provided. This would also be pretty important given the volatility of the servers in the SSM; since they aren't named (in other words, reasonably static) groups, I think it'd be helpful to keep the association of which SSM actions go with which servers. We could even throw a quick status near the SSM "Manage" and "Clear" buttons to indicate how many actions are currently in progress, so as the user continues to navigate around they have a quick indicator of if an action completes. Outside of what we track for a request, the big question is how to thread off the work. One possibility is taskomatic, which I'm not entirely familiar with. I do see from the wiki that we have tasks scheduled for once a minute, which I think is an acceptable delay before one of these SSM actions begins to process. I think we have to be able to allow more than one of these SSM actions to take place concurrently, however there is a definite upper limit to how many we should allow at once. We'll also need a mechanism to time out these actions and potentially a way for a user-initiated cancel. I'm not sure if it's possible to exercise this sort of set up through taskomatic/quartz. Anyone familiar care to comment? Otherwise, outside of EJBs, I'm not sure of any other options (besides just spinning off my own threads, but I doubt we want to go that route). Thoughts? - J -- Jason Dobies (jason.dob...@redhat.com) RHN Satellite & Spacewalk RHCE# 805008743336126 Freenode: jdob @ #spacewalk, #spacewalk-devel ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
