Hello, I've been trying to use a custom appender with slf4j and either log4j, logback or `java.util.logging` as a backend (I am not picky), to report log entries above a certain level to a third party service.
I can set a custom appender in those backends, and that works fine when submitting the jobs. However, no matter what I do, at runtime in the Dataflow pipeline workers, all my slf4j loggers get replaced by `java.util.logging.Logger` loggers, with a single handler set to `com.google.cloud.dataflow.worker.runners.worker.logging.DataflowWorkerLoggingHandler`. I have not found a good workaround. I can manually add my `java.util.logging` handler for the loggers I create, but this is a bit hacky and I don't get all the loggers related to the pipeline worker itself. For instance, if an exception and `UserCodeException` are thrown, the error is logged by the pipeline worker and my custom appender/handler won't apply. This is more about some undocumented internals of the Dataflow pipeline worker, so this may be slightly out of scope for this mailer, but Google Support couldn't help and I am curious to see if anyone has some experience with that. Thanks!
