Hi,
Thank you so much for your response. I tried by specifying a temp dataset
using the temp_dataset parameter of ReadFromBigQuery and it worked.
I was looking at the BigQuerySource class but could not find any such
parameters for setting temp_dataset and my job fails by throwing same
permission e
We had a team try to use the "slowly updating global window side inputs"
pattern (on dataflow) to update some metadata in their pipeline every
minute, but surprisingly ran into errors that the side input PCollection
contained more than one element, [1] although this only manifested
intermittently.
I also did not succeed in making this pattern work some time ago. In the
link below there's my mail thread with code example - do you have a similar
use-case?
https://lists.apache.org/thread/9l74o4vqbtfgc5vkj9qq0xofffmtxswc
Will keep watching this thread for insights.
Best Regards,
Pavel Solomin
Does "Repeatedly.forever(AfterPane.elementCountAtLeast(1)" solve this? At
least in my tests it seems like this correctly only emits a single element
per pane, but I'm not sure how much of a guarantee there actually is that
there will never be more than N elements in a pane when
elementCountAtLeast
I also had issues using this pattern. In most cases, it works fine, but the
duplicate error showed up after 4000 or so triggers using a 30-second
timer. I've tried to apply aggregation before View Singleton to enforce a
single element, but that didn't solve the issue.
Setting the timer to 5-minute
