gustavobini commented on issue #7516: URL: https://github.com/apache/incubator-devlake/issues/7516#issuecomment-2136665831
> https://github.com/apache/incubator-devlake/blob/6dca288d5e33bb55a8b98995df97ead71b76a082/backend/plugins/jenkins/tasks/build_collector.go#L143 > > does not collect data incrementally. It could use `NewStatefulApiCollectorForFinalizableEntity` but that does not support iterators. What about changing `collectMultiBranchJobApiBuilds` to call > https://github.com/apache/incubator-devlake/blob/6dca288d5e33bb55a8b98995df97ead71b76a082/backend/plugins/jenkins/tasks/build_collector.go#L72 > > for each entry in the iterator? > This could help, but > > https://github.com/apache/incubator-devlake/blob/3ae23bfdda0537e3cbe5eed17340e4fb115af1ce/backend/plugins/jenkins/tasks/build_extractor.go#L40 > > will still select all data from the raw table, including processed data. I don't know if raw tables are expected to have that data forever, or if they should be cleaned every now and then. I realized this also happens with other job types, but since the scale is much smaller, it's barely noticeable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
