Hi everyone!! I was wondering if there is an option to trigger window panes based on the size of the pane itself (rather than the number of elements).
To provide a little bit more of context we're backing up a PubSub topic into GCS with the "special" feature that, depending on the "type" of the message, the GCS destination is one or another. Messages' 'shape' published there is quite random, some of them are very frequent and small, some others very big but sparse... We have around 150 messages per second (in total) and we're firing every 15 minutes and experiencing OOM errors, we've considered firing based on the number of items as well, but given the randomness of the input, I don't think it will be a final solution either. Having a trigger based on size would be great, another option would be to have a dynamic shards number for the PTransform that actually writes the files. What is your recommendation for this use case? Thanks!!
