GitHub user bossenti added a comment to the discussion: Elements Service Load Balancer
Hi @luoluoyuyu, thank you very much for your proposal! This is really outstanding and shows a lot of dedication ❤️ I already look forward to see this becoming reality and I'd be happy to support this endeavor to any extent. I was wondering how the pipeline separation is supposed to work within StreamPipes. Should user-defined pipelines be separated in "technical" pipelines under the hood? Do you already have an idea how this could look like? If not, I'm fine - we'll find out 🙂 Do you already have a rough idea how we can plan the implementation effort? Where do we want to start? Do we want to address one use case first (e.g., pipeline allocation) and then focus on the remaining? With respect to the actual implementation: Do we want/need to build everything from scratch? Are there some libraries/frameworks we may adopt? Does it make sense to have a look how other tools handle load balancing of DAG execution? Maybe something like Airflow? How does pipeline separation work? GitHub link: https://github.com/apache/streampipes/discussions/2552#discussioncomment-8782921 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
