CarlosJuncher03 commented on issue #8563: URL: https://github.com/apache/hop/issues/8563#issuecomment-5813783172
Yes, I agree with your point. In my case, the limit was reached because one pipeline processes paginated JSON responses from an API. My intention with Execution Information is not to store all processed data, but mainly to keep a small preview of executions for troubleshooting. For example, if there is a business rule issue or a bug later, having a sample of the data from that execution can help understand what happened. The difficulty I found is that the configuration is applied at the execution metadata level, so it affects all pipelines. I also tried storing the execution information in files using an S3 URI, but that had a noticeable impact on pipeline performance. Maybe one possible option would be to allow data collection to be disabled or configured per pipeline/transform. That way, pipelines that produce large payloads could be excluded while keeping the execution preview for the others. Just an idea — I understand that increasing the Jackson limit itself may not be the best solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
