Not exactly sure what your use case is. This year, at our Beam Summit, Shunping talked about Beam State and OrderedListStates: https://beamsummit.org/sessions/2024/introducing-ordered-list-states/. This might be helpful for you.
On Sat, Sep 28, 2024 at 10:30 AM Settara Pramod <pramodsettar...@gmail.com> wrote: > Hi Apache Beam Dev Team, > > First of all, thank you for developing such an amazing project and making > it open-source. > > I have a use case where I encountered some limitations in using Apache > Beam to solve my problem. I am working with workloads that are tied to > specific sequences. My goal is to process workloads in order, where > workloads with the same sequence should run concurrently, but workloads in > the next sequence must wait until the previous sequence has fully completed. > > With Apache Beam, I found it difficult to achieve this balance. I was only > able to process workloads either sequentially by enforcing strict ordering > or run all workloads in parallel. Neither approach fully met my needs. > > I considered a workaround where one pipeline calls another, but I believe > this would be inefficient and resource-intensive, leading to poor design > and scalability. > > Is there a way to extend or customize Apache Beam to handle this use case > more effectively? I would greatly appreciate any guidance or suggestions on > this. > > I would also be happy to set up a call if further clarification is needed > regarding my use case. > > Thanks & Regards > Pramod > >
