klesh commented on issue #5849: URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1677068107
@renflo I'm afraid it is not that simple. Pagination has nothing to do with the Incremental Sync, in fact, it is creating problems for the purpose. Keep in mind jobs are listed in Reverse Order, and not only do we have to fetch the new jobs, but also to update the existing records. Consider the following flow: 1. On the 1st day, we collect 2 pages, 10 records in total from the data source. 9 of them are succeeded or failed while one of them is **RUNNING**, all good. 2. On the 2nd day, 3 new records have been added, they appear on First page at 0-2 indexes. So, what devlake is doing: 1. Fetch the First Page, extract the new records, and stop fetching the next page. 2. Iterate all Unfinished Jobs in the database and try to fetch them again to cover what was left from the previous step. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
