klesh commented on issue #5849: URL: https://github.com/apache/incubator-devlake/issues/5849#issuecomment-1674191121
@renflo 1. Incremental collection support is listed via https://devlake.apache.org/docs/Overview/SupportedDataSources , I would like to avoid full collection, but it is highly dependent on the API capabilities, e.g. API have to support filtering records by time or sorting by it, etc. 2. Seems like you were running multiple pipelines in parallel to collect data from the same Gitlab server, it would certainly exceed the rate limit of the server since Rate Limit control is under the scope of a single pipeline at this point. We haven't figured out an elegant way to handle the rate limit over multiple concurrent pipelines yet, and most importantly, collecting the same GitLab server in parallel would not speed things up due to the Rate Limit, I think a better solution is to make them run in Sequential Order, and of course, it not available at this point, I suggest you set different times for each of them for the time being. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
