GitHub user dmainou edited a comment on the discussion: I/O Error - Database LookUp transform failing
Hey, You seem to be wanting to do way too much in one pipeline. Too many steps to understand what's happening. Potentially too many steps to have enough cores and memory to allocate. Potentially too many database connection competing for resources. Start small, keep it small, fail fast and move on. Your screenshot, could benefit from some artistic layout that enables understanding what's happening. Looking at it makes me think that perhaps it would be better executed as a Workflow that has many small pipelines. Each performing a small piece of the logic with either a DB or a serialised file input. With the exception of the last pipeline, each of them outputting to a serialised file. Break it down into workable bits. GitHub link: https://github.com/apache/hop/discussions/5939#discussioncomment-14863895 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
