The trigger definition in the sample code you have is using discarding firing mode. Try swapping to using accumulating mode.
On Thu, May 31, 2018 at 1:42 AM Carlos Alonso <[email protected]> wrote: > But I think what I'm experiencing is quite different. Basically the side > input is updated, but only one element is found on the Iterable that is the > value of any key of the multimap. > > I mean, no concatenation seems to be happening. On the linked thread, Kenn > suggests that every firing will add the new value to the set of values for > the emitted key, but what I'm experiencing is that the new value is there, > but just itself (i.e: is the only element in the set). > > @Robert, I'm using > Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane()) > > On Wed, May 30, 2018 at 7:46 PM Lukasz Cwik <[email protected]> wrote: > >> An alternative to the thread that Kenn linked (adding support for >> retractions) is to add explicit support for combiners into side inputs. The >> system currently works by using a hardcoded concatenating combiner, so >> maps, lists, iterables, singletons, multimaps all work by concatenating the >> set of values emitted and then turning it into a view which is why it is an >> error for a singleton and map view if the trigger fires multiple times. >> >> On Wed, May 30, 2018 at 10:01 AM Kenneth Knowles <[email protected]> wrote: >> >>> Yes, this is a known issue. Here's a prior discussion: >>> https://lists.apache.org/thread.html/e9518f5d5f4bcf7bab02de2cb9fe1bd5293d87aa12d46de1eac4600b@%3Cuser.beam.apache.org%3E >>> >>> It is actually long-standing and the solution is known but hard. >>> >>> >>> >>> On Wed, May 30, 2018 at 9:48 AM Carlos Alonso <[email protected]> >>> wrote: >>> >>>> Hi everyone!! >>>> >>>> Working with multimap based side inputs on the global window I'm >>>> experiencing something unexpected (at least to me) that I'd like to share >>>> with you to clarify. >>>> >>>> The way I understand multimaps is that when one emits two values for >>>> the same key for the same window (obvious thing here as I'm working on the >>>> Global one), the newly emitted values are appended to the Iterable >>>> collection that is the value for that particular key on the map. >>>> >>>> Testing it in this job (it is using scio, but side inputs are >>>> implemented with PCollectionViews): >>>> https://github.com/calonso/beam_experiments/blob/master/refreshingsideinput/src/main/scala/com/mrcalonso/RefreshingSideInput2.scala >>>> >>>> The steps to reproduce are: >>>> 1. Create one table on the target BQ >>>> 2. Run the job >>>> 3. Patch the table on BQ (add one field), this should generate a new >>>> TableSchema for the corresponding TableReference >>>> 4. An updated value of the fields number appear on the logs, but there >>>> is only one element within the iterable, as if it had been updated instead >>>> of appended!! >>>> >>>> Is that the expected behaviour? Is a bug? Am I missing something? >>>> >>>> Thanks! >>>> >>>
