Hi Alex, I'll try to give a quick overview of how jobs-service schedules jobs to be executed, and I believe there might be some options to achieve what you described.
A request for a new job is received through the REST or Event APIs, and once the request is received the flow to schedule a job is executed (let's call scheduling process) and the job is persisted at some point. The way job service is currently implemented it keeps in-memory only a "bulk" or a "set" of jobs to avoid having a huge amount of unnecessary jobs in-memory, so it is kept only jobs that are supposed to be executed in the next time window and the size of this window is configurable, for example 10 minutes. So, the request of a new job executes the scheduling process and if the job is not in the next time window it just persists in DB and waits to be scheduled at some time. Ok, here is the other point where the scheduling process is executed, there is a background task that basically executes polling in DB to check jobs to be executed in the current time window and executes the scheduling process for these jobs fetched from DB. So, the simplest alternative to achieve the use case you described, would be to have an addon in runtimes where once a timer node is executed it should basically create a new job (in the schema used by job-service) and persist in job-service DB, this way that background task would fetch for jobs polling the DB as I described before, to facilitate this, the repository classes responsible to persist jobs could be externalized as a module/lib to be used by the addon in the runtimes. The other option would be to externalize as a lib with the core that contains the scheduling process + the repository of jobs-service, but THB the value of this would be just to skip the polling interval of the background task in the first polling. Anyway, I hope it helps in the discussion, and sorry for the long text. _____________________________________ *Tiago Marchetti Dolphine* <[email protected]> On Mon, Nov 13, 2023 at 9:16 AM Alex Porcelli <[email protected]> wrote: > I'm writing to initiate a discussion about our Job Service's current > architecture and its implications on the simplified architecture > approach. > > Context Setting > The simplified architecture we're striving for focuses on centralizing > database write operations. This design ensures data consistency by > having a single point for write operations. In contrast, data actions > and queries are handled by distributed components, primarily focused > on reading data. > > The Problem > Our current architecture in the Job Service deviates from this ideal. > It relies heavily on a fully distributed implementation, using runtime > to invoke remote REST calls. This approach is not in alignment with > our objective of a simplified architecture, where writes should be > centralized. > > The Point > To realign with the simplified architecture goals, the Job Service > needs a restructure similar to what has been implemented in Data > Index. This change would involve co-locating the write operations with > the runtime application. The execution of jobs would then be managed > by a separate component, maintaining the integrity of the simplified > architecture. > > I'm keen to hear your thoughts on this topic, any potential challenges > you foresee, and suggestions for implementation. > > Looking forward to a fruitful discussion. > > - > Alex > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
