+1 to the second option... As data index works like that. El lun, 13 nov 2023, 13:54, Tiago Marchetti Dolphine < [email protected]> escribió:
> Hi Alex, > I'll try to give a quick overview of how jobs-service schedules jobs to be > executed, and I believe there might be some options to achieve what you > described. > > A request for a new job is received through the REST or Event APIs, and > once the request is received the flow to schedule a job is executed (let's > call scheduling process) and the job is persisted at some point. > The way job service is currently implemented it keeps in-memory only a > "bulk" or a "set" of jobs to avoid having a huge amount of unnecessary jobs > in-memory, so it is kept only jobs that are supposed to be executed in the > next time window and the size of this window is configurable, for example > 10 minutes. > So, the request of a new job executes the scheduling process and if the job > is not in the next time window it just persists in DB and waits to be > scheduled at some time. > Ok, here is the other point where the scheduling process is executed, there > is a background task that basically executes polling in DB to check jobs to > be executed in the current time window and executes the scheduling process > for these jobs fetched from DB. > > So, the simplest alternative to achieve the use case you described, would > be to have an addon in runtimes where once a timer node is executed it > should basically create a new job (in the schema used by job-service) and > persist in job-service DB, this way that background task would fetch for > jobs polling the DB as I described before, to facilitate this, the > repository classes responsible to persist jobs could be externalized as a > module/lib to be used by the addon in the runtimes. > > The other option would be to externalize as a lib with the core that > contains the scheduling process + the repository of jobs-service, but THB > the value of this would be just to skip the polling interval of the > background task in the first polling. > > Anyway, I hope it helps in the discussion, and sorry for the long text. > _____________________________________ > *Tiago Marchetti Dolphine* > > <[email protected]> > > > On Mon, Nov 13, 2023 at 9:16 AM Alex Porcelli <[email protected]> wrote: > > > I'm writing to initiate a discussion about our Job Service's current > > architecture and its implications on the simplified architecture > > approach. > > > > Context Setting > > The simplified architecture we're striving for focuses on centralizing > > database write operations. This design ensures data consistency by > > having a single point for write operations. In contrast, data actions > > and queries are handled by distributed components, primarily focused > > on reading data. > > > > The Problem > > Our current architecture in the Job Service deviates from this ideal. > > It relies heavily on a fully distributed implementation, using runtime > > to invoke remote REST calls. This approach is not in alignment with > > our objective of a simplified architecture, where writes should be > > centralized. > > > > The Point > > To realign with the simplified architecture goals, the Job Service > > needs a restructure similar to what has been implemented in Data > > Index. This change would involve co-locating the write operations with > > the runtime application. The execution of jobs would then be managed > > by a separate component, maintaining the integrity of the simplified > > architecture. > > > > I'm keen to hear your thoughts on this topic, any potential challenges > > you foresee, and suggestions for implementation. > > > > Looking forward to a fruitful discussion. > > > > - > > Alex > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
