Rookie625 commented on issue #7752: URL: https://github.com/apache/hop/issues/7752#issuecomment-5234790367
> It's not really a bug but a documented behavior with circular dependencies. Loading all data into memory is perhaps less than ideal in all scenario. There's a reason why we don't have infinite buffers between transforms and that is obviously to avoid running out of memory. > > That being said, if we could detect the circular dependency we could opt to serialize rows to disk to get around the limitation as a generic solution. I guess that shifts the problem to detecting that there is a circular dependency at play. > > In any case, my preference would be to avoid starting extra threads and possibly causing out of memory errors and to favor a more generic solution for the problem. Thank you for taking the time to provide such a thorough and candid explanation. I completely see your point. Looking at this from the engine's perspective, it's absolutely not a bug—it's a conscious trade-off between performance and stability. The decision to avoid infinite buffers is precisely what prevents the engine from collapsing under memory pressure, and that foundation is something I fully respect. My initial observation came from a different angle—more from the user experience side, where encountering a silent hang can be confusing. But I now realize that what feels like a "limitation" from the outside is actually a deliberate architectural safeguard from the inside. It's a classic case of two valid perspectives looking at the same problem. Your idea about detecting circular dependencies and spilling to disk as a generic solution is particularly insightful. It strikes me as an elegant long-term direction that doesn't compromise the core principle of memory safety. I also appreciate your preference for avoiding extra threads that might invite OOM errors—it reinforces that stability should always come first. This exchange has given me a much deeper appreciation for the design choices behind the scenes. Sometimes the most robust systems are the ones that refuse to compromise on their fundamental constraints, even if it means occasionally surprising the user. Thanks again for sharing your thoughts—I've learned a lot from this discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
