Hi All,
This is my first message for this maillist. Please let me know if I am
sending this message to wrong maillist.
My stream processing job are running on Google Cloud Dataflow engine. For
logging I am using Stackdriver. I added runtime slf4j-jdk14 and slf4j-api
to enable to stackdriver. Howe
The only logging options that Dataflow exposes today limit what gets logged
and not anything about how many rotated logs there are or how big they are.
All Dataflow logging options are available here:
https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/a
Thank you Lukasz for replying back to me. I have long live stream
processor. I am getting out of disk space error on dataflow worker machines
after couples days that I submitted.
As you said Dataflow worker provides very limited logging options. However
I checked code of dataflow worker's logging
There is a fluentd application that uploads the log files to Stackdriver
and then deletes the files that is separate from the Java code.
If you log way too much, you have to decide to reduce logging or block the
processing of your pipeline. You could add a root logging handler that
monitors disksp
