Hi! I am trying to adjust the log level for a Beam YAML pipeline that uses Kafka. Since Kafka is a multi-language transform, default_sdk_harness_log_level is not currently supported, which means I can't limit the logs (Kafka can be a bit too verbose sometimes). This could lead to increased logging costs if you use a cloud service such as Dataflow to run the pipeline.
Is there any initiative at the moment to support default_sdk_harness_log_level in multi-language transforms? Maybe some workaround? I suppose that in terms of Beam YAML, we could pass these options to the schema providers somehow. However, I don't think that solution is ideal, as it would not address the issue for other people using multi-language transforms outside of Beam YAML. Thanks! Ferran
