I just wanted to mention that there is quite old open issue about that: https://issues.apache.org/jira/browse/BEAM-727 <https://issues.apache.org/jira/browse/BEAM-727>
Fell free to take this one if anyone is interested. > On 2 Jan 2019, at 15:22, Juan Carlos Garcia <jcgarc...@gmail.com> wrote: > > +1 > > Am Mi., 2. Jan. 2019, 14:34 hat Abdul Qadeer <quadeer....@gmail.com > <mailto:quadeer....@gmail.com>> geschrieben: > +1 > > On Tue, 1 Jan 2019 at 12:45, <jan.d...@gmail.com <mailto:jan.d...@gmail.com>> > wrote: > +1 from my side too :-) > And ideally I would want to have some hooks to let me know the extra > partitions have been picked up (or a way to query it). > > Although if that can't be provided I can work around it myself by sending > some specific message to the partition that somewhere results in a visible > state change in the pipeline. > > Also, as a quick (semi related) heads up: I will very likely soon contribute > a change to the LogAppendTimePolicy so that the idle partition behavior > (automatic watermark generation) can be disabled. > > (of course all related to my streamy-db project) > > Kind regards, > Jan > > > On Tue, 1 Jan 2019 at 08:19, Ramesh Nethi <ramesh.ne...@gmail.com > <mailto:ramesh.ne...@gmail.com>> wrote: > +1 for this capability. This would enable pipelines to continue to run when > such changes need to be made. > > regards > Ramesh > > On Fri, 23 Nov 2018 at 00:40 Raghu Angadi <rang...@google.com > <mailto:rang...@google.com>> wrote: > On Thu, Nov 22, 2018 at 10:10 AM Raghu Angadi <rang...@google.com > <mailto:rang...@google.com>> wrote: > - New partitions will be ignored during runtime. > - Update will not succeed either. Error message on the workers should explain > the mismatch. > > This is the current state. Supporting changes to number of partition is quite > doable if there is enough user interested (even in the current > UnnoundedSource API framework). > > On Thu, Nov 22, 2018 at 2:15 AM Jozef Vilcek <jozo.vil...@gmail.com > <mailto:jozo.vil...@gmail.com>> wrote: > Hello, > just wanted to check how does Beam KafkaIO behaves when partitions are added > to the topic. > Will they be picked up or ignored during the runtime? > Will they be picked up on restart with state restore? > > Thanks, > Jozef
